Jun 02 01:38:38.328505 ip-10-0-135-117 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jun 02 01:38:38.328518 ip-10-0-135-117 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jun 02 01:38:38.328527 ip-10-0-135-117 systemd[1]: kubelet.service: Failed with result 'resources'. Jun 02 01:38:38.328917 ip-10-0-135-117 systemd[1]: Failed to start Kubernetes Kubelet. Jun 02 01:38:48.548248 ip-10-0-135-117 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jun 02 01:38:48.548269 ip-10-0-135-117 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot da19677c65da4f318a9bfcb79ab669b1 -- Jun 02 01:41:22.236410 ip-10-0-135-117 systemd[1]: Starting Kubernetes Kubelet... Jun 02 01:41:22.669995 ip-10-0-135-117 kubenswrapper[2571]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jun 02 01:41:22.669995 ip-10-0-135-117 kubenswrapper[2571]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jun 02 01:41:22.669995 ip-10-0-135-117 kubenswrapper[2571]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jun 02 01:41:22.669995 ip-10-0-135-117 kubenswrapper[2571]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jun 02 01:41:22.669995 ip-10-0-135-117 kubenswrapper[2571]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jun 02 01:41:22.671890 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.671794 2571 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jun 02 01:41:22.675640 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675625 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jun 02 01:41:22.675640 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675640 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jun 02 01:41:22.675705 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675645 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jun 02 01:41:22.675705 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675648 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jun 02 01:41:22.675705 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675651 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jun 02 01:41:22.675705 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675654 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jun 02 01:41:22.675705 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675668 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jun 02 01:41:22.675705 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675671 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jun 02 01:41:22.675705 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675674 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jun 02 01:41:22.675705 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675676 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jun 02 01:41:22.675705 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675680 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jun 02 01:41:22.675705 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675685 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jun 02 01:41:22.675705 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675688 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jun 02 01:41:22.675705 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675690 2571 feature_gate.go:328] unrecognized feature gate: Example2 Jun 02 01:41:22.675705 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675693 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jun 02 01:41:22.675705 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675695 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jun 02 01:41:22.675705 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675698 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jun 02 01:41:22.675705 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675700 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jun 02 01:41:22.675705 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675703 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jun 02 01:41:22.675705 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675705 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jun 02 01:41:22.675705 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675708 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jun 02 01:41:22.676243 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675711 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jun 02 01:41:22.676243 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675714 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jun 02 01:41:22.676243 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675717 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jun 02 01:41:22.676243 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675719 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Jun 02 01:41:22.676243 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675722 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jun 02 01:41:22.676243 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675724 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Jun 02 01:41:22.676243 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675727 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Jun 02 01:41:22.676243 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675729 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jun 02 01:41:22.676243 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675732 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jun 02 01:41:22.676243 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675734 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jun 02 01:41:22.676243 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675737 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jun 02 01:41:22.676243 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675739 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jun 02 01:41:22.676243 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675741 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jun 02 01:41:22.676243 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675744 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jun 02 01:41:22.676243 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675746 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jun 02 01:41:22.676243 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675749 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jun 02 01:41:22.676243 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675751 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jun 02 01:41:22.676243 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675763 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jun 02 01:41:22.676243 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675771 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jun 02 01:41:22.676243 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675774 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jun 02 01:41:22.677046 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675777 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jun 02 01:41:22.677046 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675779 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jun 02 01:41:22.677046 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675781 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jun 02 01:41:22.677046 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675784 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jun 02 01:41:22.677046 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675786 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jun 02 01:41:22.677046 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675789 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jun 02 01:41:22.677046 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675791 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jun 02 01:41:22.677046 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675794 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jun 02 01:41:22.677046 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675796 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jun 02 01:41:22.677046 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675799 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS Jun 02 01:41:22.677046 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675801 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jun 02 01:41:22.677046 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675803 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jun 02 01:41:22.677046 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675806 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jun 02 01:41:22.677046 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675809 2571 feature_gate.go:328] unrecognized feature gate: Example Jun 02 01:41:22.677046 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675812 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jun 02 01:41:22.677046 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675814 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jun 02 01:41:22.677046 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675817 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jun 02 01:41:22.677046 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675819 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jun 02 01:41:22.677046 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675822 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jun 02 01:41:22.677046 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675824 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jun 02 01:41:22.677558 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675827 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jun 02 01:41:22.677558 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675829 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jun 02 01:41:22.677558 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675833 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jun 02 01:41:22.677558 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675835 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jun 02 01:41:22.677558 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675838 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jun 02 01:41:22.677558 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675841 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI Jun 02 01:41:22.677558 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675843 2571 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController Jun 02 01:41:22.677558 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675846 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jun 02 01:41:22.677558 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675849 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jun 02 01:41:22.677558 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675851 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jun 02 01:41:22.677558 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675853 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jun 02 01:41:22.677558 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.675906 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jun 02 01:41:22.677558 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.676005 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Jun 02 01:41:22.677558 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.676008 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jun 02 01:41:22.677558 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.676011 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jun 02 01:41:22.677558 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.676014 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jun 02 01:41:22.677558 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.676017 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jun 02 01:41:22.677558 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.676020 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jun 02 01:41:22.677558 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.676024 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jun 02 01:41:22.677558 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.676108 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Jun 02 01:41:22.678044 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.676687 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jun 02 01:41:22.678044 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.676695 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jun 02 01:41:22.678044 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.676707 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jun 02 01:41:22.678044 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.676716 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations Jun 02 01:41:22.678044 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.676722 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jun 02 01:41:22.678044 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677506 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jun 02 01:41:22.678044 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677517 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Jun 02 01:41:22.678044 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677521 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jun 02 01:41:22.678044 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677524 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jun 02 01:41:22.678044 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677527 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jun 02 01:41:22.678044 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677529 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jun 02 01:41:22.678044 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677532 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jun 02 01:41:22.678044 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677534 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jun 02 01:41:22.678044 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677537 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jun 02 01:41:22.678044 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677539 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jun 02 01:41:22.678044 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677542 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jun 02 01:41:22.678044 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677544 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jun 02 01:41:22.678044 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677546 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jun 02 01:41:22.678044 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677549 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jun 02 01:41:22.678589 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677552 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jun 02 01:41:22.678589 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677555 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jun 02 01:41:22.678589 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677557 2571 feature_gate.go:328] unrecognized feature gate: Example Jun 02 01:41:22.678589 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677560 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jun 02 01:41:22.678589 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677562 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jun 02 01:41:22.678589 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677565 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jun 02 01:41:22.678589 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677568 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jun 02 01:41:22.678589 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677571 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS Jun 02 01:41:22.678589 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677573 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jun 02 01:41:22.678589 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677576 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jun 02 01:41:22.678589 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677579 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jun 02 01:41:22.678589 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677581 2571 feature_gate.go:328] unrecognized feature gate: Example2 Jun 02 01:41:22.678589 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677584 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jun 02 01:41:22.678589 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677586 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jun 02 01:41:22.678589 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677592 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jun 02 01:41:22.678589 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677595 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jun 02 01:41:22.678589 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677602 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jun 02 01:41:22.678589 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677604 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jun 02 01:41:22.678589 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677607 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jun 02 01:41:22.679066 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677611 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jun 02 01:41:22.679066 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677615 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jun 02 01:41:22.679066 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677618 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jun 02 01:41:22.679066 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677621 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jun 02 01:41:22.679066 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677623 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI Jun 02 01:41:22.679066 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677626 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jun 02 01:41:22.679066 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677629 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jun 02 01:41:22.679066 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677631 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jun 02 01:41:22.679066 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677634 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jun 02 01:41:22.679066 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677636 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jun 02 01:41:22.679066 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677639 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jun 02 01:41:22.679066 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677641 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jun 02 01:41:22.679066 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677644 2571 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController Jun 02 01:41:22.679066 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677647 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jun 02 01:41:22.679066 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677650 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jun 02 01:41:22.679066 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677653 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jun 02 01:41:22.679066 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677655 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jun 02 01:41:22.679066 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677658 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jun 02 01:41:22.679066 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677661 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jun 02 01:41:22.679066 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677664 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jun 02 01:41:22.679567 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677667 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jun 02 01:41:22.679567 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677670 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jun 02 01:41:22.679567 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677672 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jun 02 01:41:22.679567 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677675 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jun 02 01:41:22.679567 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677677 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jun 02 01:41:22.679567 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677680 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jun 02 01:41:22.679567 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677683 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Jun 02 01:41:22.679567 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677685 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jun 02 01:41:22.679567 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677688 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jun 02 01:41:22.679567 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677690 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jun 02 01:41:22.679567 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677693 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jun 02 01:41:22.679567 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677695 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jun 02 01:41:22.679567 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677697 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Jun 02 01:41:22.679567 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677701 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jun 02 01:41:22.679567 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677704 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jun 02 01:41:22.679567 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677707 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jun 02 01:41:22.679567 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677709 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Jun 02 01:41:22.679567 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677712 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jun 02 01:41:22.679567 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677714 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jun 02 01:41:22.680028 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677717 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jun 02 01:41:22.680028 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677719 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jun 02 01:41:22.680028 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677722 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Jun 02 01:41:22.680028 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677724 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jun 02 01:41:22.680028 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677727 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jun 02 01:41:22.680028 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677729 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jun 02 01:41:22.680028 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677732 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jun 02 01:41:22.680028 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677735 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jun 02 01:41:22.680028 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677737 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jun 02 01:41:22.680028 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677742 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jun 02 01:41:22.680028 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677746 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jun 02 01:41:22.680028 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677749 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jun 02 01:41:22.680028 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677752 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations Jun 02 01:41:22.680028 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.677754 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jun 02 01:41:22.680028 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678417 2571 flags.go:64] FLAG: --address="0.0.0.0" Jun 02 01:41:22.680028 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678427 2571 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jun 02 01:41:22.680028 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678435 2571 flags.go:64] FLAG: --anonymous-auth="true" Jun 02 01:41:22.680028 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678440 2571 flags.go:64] FLAG: --application-metrics-count-limit="100" Jun 02 01:41:22.680028 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678444 2571 flags.go:64] FLAG: --authentication-token-webhook="false" Jun 02 01:41:22.680028 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678447 2571 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jun 02 01:41:22.680028 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678452 2571 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jun 02 01:41:22.680557 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678456 2571 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jun 02 01:41:22.680557 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678459 2571 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jun 02 01:41:22.680557 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678462 2571 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jun 02 01:41:22.680557 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678466 2571 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jun 02 01:41:22.680557 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678469 2571 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jun 02 01:41:22.680557 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678473 2571 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jun 02 01:41:22.680557 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678476 2571 flags.go:64] FLAG: --cgroup-root="" Jun 02 01:41:22.680557 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678479 2571 flags.go:64] FLAG: --cgroups-per-qos="true" Jun 02 01:41:22.680557 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678481 2571 flags.go:64] FLAG: --client-ca-file="" Jun 02 01:41:22.680557 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678485 2571 flags.go:64] FLAG: --cloud-config="" Jun 02 01:41:22.680557 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678487 2571 flags.go:64] FLAG: --cloud-provider="external" Jun 02 01:41:22.680557 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678491 2571 flags.go:64] FLAG: --cluster-dns="[]" Jun 02 01:41:22.680557 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678495 2571 flags.go:64] FLAG: --cluster-domain="" Jun 02 01:41:22.680557 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678498 2571 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jun 02 01:41:22.680557 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678501 2571 flags.go:64] FLAG: --config-dir="" Jun 02 01:41:22.680557 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678504 2571 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jun 02 01:41:22.680557 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678507 2571 flags.go:64] FLAG: --container-log-max-files="5" Jun 02 01:41:22.680557 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678512 2571 flags.go:64] FLAG: --container-log-max-size="10Mi" Jun 02 01:41:22.680557 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678515 2571 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jun 02 01:41:22.680557 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678518 2571 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jun 02 01:41:22.680557 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678521 2571 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jun 02 01:41:22.680557 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678524 2571 flags.go:64] FLAG: --contention-profiling="false" Jun 02 01:41:22.680557 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678527 2571 flags.go:64] FLAG: --cpu-cfs-quota="true" Jun 02 01:41:22.680557 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678530 2571 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jun 02 01:41:22.681146 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678533 2571 flags.go:64] FLAG: --cpu-manager-policy="none" Jun 02 01:41:22.681146 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678536 2571 flags.go:64] FLAG: --cpu-manager-policy-options="" Jun 02 01:41:22.681146 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678540 2571 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jun 02 01:41:22.681146 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678543 2571 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jun 02 01:41:22.681146 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678545 2571 flags.go:64] FLAG: --enable-debugging-handlers="true" Jun 02 01:41:22.681146 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678548 2571 flags.go:64] FLAG: --enable-load-reader="false" Jun 02 01:41:22.681146 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678551 2571 flags.go:64] FLAG: --enable-server="true" Jun 02 01:41:22.681146 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678554 2571 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jun 02 01:41:22.681146 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678558 2571 flags.go:64] FLAG: --event-burst="100" Jun 02 01:41:22.681146 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678562 2571 flags.go:64] FLAG: --event-qps="50" Jun 02 01:41:22.681146 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678565 2571 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jun 02 01:41:22.681146 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678568 2571 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jun 02 01:41:22.681146 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678571 2571 flags.go:64] FLAG: --eviction-hard="" Jun 02 01:41:22.681146 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678575 2571 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jun 02 01:41:22.681146 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678578 2571 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jun 02 01:41:22.681146 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678581 2571 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jun 02 01:41:22.681146 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678584 2571 flags.go:64] FLAG: --eviction-soft="" Jun 02 01:41:22.681146 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678587 2571 flags.go:64] FLAG: --eviction-soft-grace-period="" Jun 02 01:41:22.681146 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678590 2571 flags.go:64] FLAG: --exit-on-lock-contention="false" Jun 02 01:41:22.681146 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678593 2571 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jun 02 01:41:22.681146 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678596 2571 flags.go:64] FLAG: --experimental-mounter-path="" Jun 02 01:41:22.681146 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678599 2571 flags.go:64] FLAG: --fail-cgroupv1="false" Jun 02 01:41:22.681146 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678602 2571 flags.go:64] FLAG: --fail-swap-on="true" Jun 02 01:41:22.681146 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678605 2571 flags.go:64] FLAG: --feature-gates="" Jun 02 01:41:22.681146 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678609 2571 flags.go:64] FLAG: --file-check-frequency="20s" Jun 02 01:41:22.681783 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678613 2571 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jun 02 01:41:22.681783 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678616 2571 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jun 02 01:41:22.681783 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678619 2571 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jun 02 01:41:22.681783 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678622 2571 flags.go:64] FLAG: --healthz-port="10248" Jun 02 01:41:22.681783 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678625 2571 flags.go:64] FLAG: --help="false" Jun 02 01:41:22.681783 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678628 2571 flags.go:64] FLAG: --hostname-override="ip-10-0-135-117.ec2.internal" Jun 02 01:41:22.681783 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678631 2571 flags.go:64] FLAG: --housekeeping-interval="10s" Jun 02 01:41:22.681783 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678634 2571 flags.go:64] FLAG: --http-check-frequency="20s" Jun 02 01:41:22.681783 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678636 2571 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jun 02 01:41:22.681783 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678640 2571 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jun 02 01:41:22.681783 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678643 2571 flags.go:64] FLAG: --image-gc-high-threshold="85" Jun 02 01:41:22.681783 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678646 2571 flags.go:64] FLAG: --image-gc-low-threshold="80" Jun 02 01:41:22.681783 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678649 2571 flags.go:64] FLAG: --image-service-endpoint="" Jun 02 01:41:22.681783 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678651 2571 flags.go:64] FLAG: --kernel-memcg-notification="false" Jun 02 01:41:22.681783 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678654 2571 flags.go:64] FLAG: --kube-api-burst="100" Jun 02 01:41:22.681783 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678657 2571 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jun 02 01:41:22.681783 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678660 2571 flags.go:64] FLAG: --kube-api-qps="50" Jun 02 01:41:22.681783 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678664 2571 flags.go:64] FLAG: --kube-reserved="" Jun 02 01:41:22.681783 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678667 2571 flags.go:64] FLAG: --kube-reserved-cgroup="" Jun 02 01:41:22.681783 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678670 2571 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jun 02 01:41:22.681783 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678673 2571 flags.go:64] FLAG: --kubelet-cgroups="" Jun 02 01:41:22.681783 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678677 2571 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jun 02 01:41:22.681783 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678680 2571 flags.go:64] FLAG: --lock-file="" Jun 02 01:41:22.681783 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678683 2571 flags.go:64] FLAG: --log-cadvisor-usage="false" Jun 02 01:41:22.682372 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678686 2571 flags.go:64] FLAG: --log-flush-frequency="5s" Jun 02 01:41:22.682372 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678690 2571 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jun 02 01:41:22.682372 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678700 2571 flags.go:64] FLAG: --log-json-split-stream="false" Jun 02 01:41:22.682372 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678703 2571 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jun 02 01:41:22.682372 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678706 2571 flags.go:64] FLAG: --log-text-split-stream="false" Jun 02 01:41:22.682372 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678709 2571 flags.go:64] FLAG: --logging-format="text" Jun 02 01:41:22.682372 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678712 2571 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jun 02 01:41:22.682372 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678716 2571 flags.go:64] FLAG: --make-iptables-util-chains="true" Jun 02 01:41:22.682372 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678718 2571 flags.go:64] FLAG: --manifest-url="" Jun 02 01:41:22.682372 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678721 2571 flags.go:64] FLAG: --manifest-url-header="" Jun 02 01:41:22.682372 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678726 2571 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jun 02 01:41:22.682372 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678729 2571 flags.go:64] FLAG: --max-open-files="1000000" Jun 02 01:41:22.682372 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678732 2571 flags.go:64] FLAG: --max-pods="110" Jun 02 01:41:22.682372 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678735 2571 flags.go:64] FLAG: --maximum-dead-containers="-1" Jun 02 01:41:22.682372 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678739 2571 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jun 02 01:41:22.682372 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678742 2571 flags.go:64] FLAG: --memory-manager-policy="None" Jun 02 01:41:22.682372 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678745 2571 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jun 02 01:41:22.682372 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678748 2571 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jun 02 01:41:22.682372 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678751 2571 flags.go:64] FLAG: --node-ip="0.0.0.0" Jun 02 01:41:22.682372 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678754 2571 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jun 02 01:41:22.682372 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678761 2571 flags.go:64] FLAG: --node-status-max-images="50" Jun 02 01:41:22.682372 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678764 2571 flags.go:64] FLAG: --node-status-update-frequency="10s" Jun 02 01:41:22.682372 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678771 2571 flags.go:64] FLAG: --oom-score-adj="-999" Jun 02 01:41:22.682372 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678774 2571 flags.go:64] FLAG: --pod-cidr="" Jun 02 01:41:22.682951 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678777 2571 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f1083dc6abd9805f79797fa11c9accacc4a38caa073c94cb99097faffce05983" Jun 02 01:41:22.682951 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678782 2571 flags.go:64] FLAG: --pod-manifest-path="" Jun 02 01:41:22.682951 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678785 2571 flags.go:64] FLAG: --pod-max-pids="-1" Jun 02 01:41:22.682951 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678788 2571 flags.go:64] FLAG: --pods-per-core="0" Jun 02 01:41:22.682951 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678798 2571 flags.go:64] FLAG: --port="10250" Jun 02 01:41:22.682951 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678804 2571 flags.go:64] FLAG: --protect-kernel-defaults="false" Jun 02 01:41:22.682951 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678807 2571 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e9c587547df6ec83" Jun 02 01:41:22.682951 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678810 2571 flags.go:64] FLAG: --qos-reserved="" Jun 02 01:41:22.682951 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678813 2571 flags.go:64] FLAG: --read-only-port="10255" Jun 02 01:41:22.682951 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678816 2571 flags.go:64] FLAG: --register-node="true" Jun 02 01:41:22.682951 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678819 2571 flags.go:64] FLAG: --register-schedulable="true" Jun 02 01:41:22.682951 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678822 2571 flags.go:64] FLAG: --register-with-taints="" Jun 02 01:41:22.682951 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678825 2571 flags.go:64] FLAG: --registry-burst="10" Jun 02 01:41:22.682951 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678829 2571 flags.go:64] FLAG: --registry-qps="5" Jun 02 01:41:22.682951 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678831 2571 flags.go:64] FLAG: --reserved-cpus="" Jun 02 01:41:22.682951 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678834 2571 flags.go:64] FLAG: --reserved-memory="" Jun 02 01:41:22.682951 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678837 2571 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jun 02 01:41:22.682951 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678840 2571 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jun 02 01:41:22.682951 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678843 2571 flags.go:64] FLAG: --rotate-certificates="false" Jun 02 01:41:22.682951 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678846 2571 flags.go:64] FLAG: --rotate-server-certificates="false" Jun 02 01:41:22.682951 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678850 2571 flags.go:64] FLAG: --runonce="false" Jun 02 01:41:22.682951 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678853 2571 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jun 02 01:41:22.682951 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678856 2571 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jun 02 01:41:22.682951 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678859 2571 flags.go:64] FLAG: --seccomp-default="false" Jun 02 01:41:22.682951 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678862 2571 flags.go:64] FLAG: --serialize-image-pulls="true" Jun 02 01:41:22.683571 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678865 2571 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jun 02 01:41:22.683571 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678868 2571 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jun 02 01:41:22.683571 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678871 2571 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jun 02 01:41:22.683571 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678874 2571 flags.go:64] FLAG: --storage-driver-password="root" Jun 02 01:41:22.683571 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678877 2571 flags.go:64] FLAG: --storage-driver-secure="false" Jun 02 01:41:22.683571 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678880 2571 flags.go:64] FLAG: --storage-driver-table="stats" Jun 02 01:41:22.683571 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678884 2571 flags.go:64] FLAG: --storage-driver-user="root" Jun 02 01:41:22.683571 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678887 2571 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jun 02 01:41:22.683571 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678890 2571 flags.go:64] FLAG: --sync-frequency="1m0s" Jun 02 01:41:22.683571 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678893 2571 flags.go:64] FLAG: --system-cgroups="" Jun 02 01:41:22.683571 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678896 2571 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jun 02 01:41:22.683571 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678901 2571 flags.go:64] FLAG: --system-reserved-cgroup="" Jun 02 01:41:22.683571 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678904 2571 flags.go:64] FLAG: --tls-cert-file="" Jun 02 01:41:22.683571 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678907 2571 flags.go:64] FLAG: --tls-cipher-suites="[]" Jun 02 01:41:22.683571 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678911 2571 flags.go:64] FLAG: --tls-min-version="" Jun 02 01:41:22.683571 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678914 2571 flags.go:64] FLAG: --tls-private-key-file="" Jun 02 01:41:22.683571 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678916 2571 flags.go:64] FLAG: --topology-manager-policy="none" Jun 02 01:41:22.683571 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678919 2571 flags.go:64] FLAG: --topology-manager-policy-options="" Jun 02 01:41:22.683571 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678922 2571 flags.go:64] FLAG: --topology-manager-scope="container" Jun 02 01:41:22.683571 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678924 2571 flags.go:64] FLAG: --v="2" Jun 02 01:41:22.683571 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678931 2571 flags.go:64] FLAG: --version="false" Jun 02 01:41:22.683571 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678935 2571 flags.go:64] FLAG: --vmodule="" Jun 02 01:41:22.683571 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678939 2571 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jun 02 01:41:22.683571 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.678943 2571 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jun 02 01:41:22.683571 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679047 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jun 02 01:41:22.684193 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679052 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jun 02 01:41:22.684193 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679056 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jun 02 01:41:22.684193 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679060 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jun 02 01:41:22.684193 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679063 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jun 02 01:41:22.684193 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679065 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jun 02 01:41:22.684193 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679070 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jun 02 01:41:22.684193 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679073 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jun 02 01:41:22.684193 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679075 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jun 02 01:41:22.684193 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679078 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jun 02 01:41:22.684193 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679081 2571 feature_gate.go:328] unrecognized feature gate: Example2 Jun 02 01:41:22.684193 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679084 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jun 02 01:41:22.684193 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679086 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jun 02 01:41:22.684193 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679089 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Jun 02 01:41:22.684193 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679092 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jun 02 01:41:22.684193 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679095 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jun 02 01:41:22.684193 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679097 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jun 02 01:41:22.684193 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679100 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jun 02 01:41:22.684193 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679103 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jun 02 01:41:22.684193 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679105 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jun 02 01:41:22.684193 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679108 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations Jun 02 01:41:22.684695 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679110 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jun 02 01:41:22.684695 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679113 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jun 02 01:41:22.684695 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679116 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jun 02 01:41:22.684695 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679118 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jun 02 01:41:22.684695 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679121 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jun 02 01:41:22.684695 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679123 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jun 02 01:41:22.684695 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679140 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jun 02 01:41:22.684695 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679143 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jun 02 01:41:22.684695 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679145 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jun 02 01:41:22.684695 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679148 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jun 02 01:41:22.684695 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679150 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jun 02 01:41:22.684695 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679153 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jun 02 01:41:22.684695 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679156 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jun 02 01:41:22.684695 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679158 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jun 02 01:41:22.684695 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679161 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jun 02 01:41:22.684695 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679164 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jun 02 01:41:22.684695 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679166 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jun 02 01:41:22.684695 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679170 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Jun 02 01:41:22.684695 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679172 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jun 02 01:41:22.685174 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679175 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jun 02 01:41:22.685174 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679178 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jun 02 01:41:22.685174 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679180 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jun 02 01:41:22.685174 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679183 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jun 02 01:41:22.685174 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679185 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jun 02 01:41:22.685174 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679188 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jun 02 01:41:22.685174 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679190 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jun 02 01:41:22.685174 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679193 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS Jun 02 01:41:22.685174 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679196 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jun 02 01:41:22.685174 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679198 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jun 02 01:41:22.685174 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679201 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jun 02 01:41:22.685174 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679204 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Jun 02 01:41:22.685174 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679206 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jun 02 01:41:22.685174 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679209 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jun 02 01:41:22.685174 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679211 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI Jun 02 01:41:22.685174 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679214 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jun 02 01:41:22.685174 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679217 2571 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController Jun 02 01:41:22.685174 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679220 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jun 02 01:41:22.685174 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679222 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jun 02 01:41:22.685174 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679225 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jun 02 01:41:22.686063 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679228 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jun 02 01:41:22.686063 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679231 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jun 02 01:41:22.686063 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679233 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jun 02 01:41:22.686063 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679236 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jun 02 01:41:22.686063 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679238 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jun 02 01:41:22.686063 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679241 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jun 02 01:41:22.686063 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679243 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jun 02 01:41:22.686063 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679246 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jun 02 01:41:22.686063 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679248 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jun 02 01:41:22.686063 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679251 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jun 02 01:41:22.686063 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679255 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jun 02 01:41:22.686063 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679257 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jun 02 01:41:22.686063 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679260 2571 feature_gate.go:328] unrecognized feature gate: Example Jun 02 01:41:22.686063 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679264 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jun 02 01:41:22.686063 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679267 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jun 02 01:41:22.686063 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679270 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jun 02 01:41:22.686063 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679273 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Jun 02 01:41:22.686063 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679275 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jun 02 01:41:22.686063 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679278 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Jun 02 01:41:22.686063 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679281 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jun 02 01:41:22.686950 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679283 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jun 02 01:41:22.686950 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679286 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jun 02 01:41:22.686950 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679288 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jun 02 01:41:22.686950 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679291 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jun 02 01:41:22.686950 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679293 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jun 02 01:41:22.686950 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.679296 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jun 02 01:41:22.686950 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.680138 2571 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]} Jun 02 01:41:22.688190 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.688170 2571 server.go:530] "Kubelet version" kubeletVersion="v1.33.12" Jun 02 01:41:22.688190 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.688191 2571 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jun 02 01:41:22.688311 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688261 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jun 02 01:41:22.688311 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688269 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jun 02 01:41:22.688311 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688275 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Jun 02 01:41:22.688311 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688280 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jun 02 01:41:22.688311 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688284 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Jun 02 01:41:22.688311 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688289 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jun 02 01:41:22.688311 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688294 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jun 02 01:41:22.688311 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688298 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jun 02 01:41:22.688311 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688303 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jun 02 01:41:22.688311 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688307 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jun 02 01:41:22.688311 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688311 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jun 02 01:41:22.688311 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688315 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jun 02 01:41:22.688837 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688320 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Jun 02 01:41:22.688837 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688325 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jun 02 01:41:22.688837 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688330 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jun 02 01:41:22.688837 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688334 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jun 02 01:41:22.688837 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688338 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jun 02 01:41:22.688837 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688342 2571 feature_gate.go:328] unrecognized feature gate: Example2 Jun 02 01:41:22.688837 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688347 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jun 02 01:41:22.688837 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688351 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jun 02 01:41:22.688837 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688355 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jun 02 01:41:22.688837 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688359 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jun 02 01:41:22.688837 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688363 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jun 02 01:41:22.688837 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688367 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jun 02 01:41:22.688837 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688371 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jun 02 01:41:22.688837 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688375 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jun 02 01:41:22.688837 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688379 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jun 02 01:41:22.688837 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688383 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jun 02 01:41:22.688837 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688388 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jun 02 01:41:22.688837 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688392 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jun 02 01:41:22.688837 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688397 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jun 02 01:41:22.688837 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688403 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jun 02 01:41:22.689602 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688407 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jun 02 01:41:22.689602 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688412 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jun 02 01:41:22.689602 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688416 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations Jun 02 01:41:22.689602 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688421 2571 feature_gate.go:328] unrecognized feature gate: Example Jun 02 01:41:22.689602 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688425 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jun 02 01:41:22.689602 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688429 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jun 02 01:41:22.689602 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688433 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jun 02 01:41:22.689602 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688438 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jun 02 01:41:22.689602 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688442 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jun 02 01:41:22.689602 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688448 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jun 02 01:41:22.689602 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688454 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jun 02 01:41:22.689602 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688459 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Jun 02 01:41:22.689602 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688463 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jun 02 01:41:22.689602 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688468 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Jun 02 01:41:22.689602 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688473 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jun 02 01:41:22.689602 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688477 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jun 02 01:41:22.689602 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688481 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jun 02 01:41:22.689602 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688485 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jun 02 01:41:22.689602 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688489 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jun 02 01:41:22.689602 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688493 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jun 02 01:41:22.690111 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688498 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jun 02 01:41:22.690111 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688501 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jun 02 01:41:22.690111 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688506 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jun 02 01:41:22.690111 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688510 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jun 02 01:41:22.690111 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688514 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jun 02 01:41:22.690111 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688518 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jun 02 01:41:22.690111 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688523 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jun 02 01:41:22.690111 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688527 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS Jun 02 01:41:22.690111 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688531 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jun 02 01:41:22.690111 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688535 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jun 02 01:41:22.690111 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688539 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI Jun 02 01:41:22.690111 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688543 2571 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController Jun 02 01:41:22.690111 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688548 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jun 02 01:41:22.690111 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688553 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jun 02 01:41:22.690111 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688558 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jun 02 01:41:22.690111 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688562 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jun 02 01:41:22.690111 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688566 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jun 02 01:41:22.690111 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688571 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jun 02 01:41:22.690111 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688575 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jun 02 01:41:22.690111 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688579 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jun 02 01:41:22.690750 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688583 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jun 02 01:41:22.690750 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688587 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jun 02 01:41:22.690750 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688592 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jun 02 01:41:22.690750 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688596 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jun 02 01:41:22.690750 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688601 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jun 02 01:41:22.690750 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688609 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jun 02 01:41:22.690750 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688615 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jun 02 01:41:22.690750 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688619 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jun 02 01:41:22.690750 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688624 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jun 02 01:41:22.690750 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688629 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jun 02 01:41:22.690750 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688633 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jun 02 01:41:22.690750 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688637 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jun 02 01:41:22.690750 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688641 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jun 02 01:41:22.690750 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688645 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jun 02 01:41:22.690750 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.688654 2571 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]} Jun 02 01:41:22.691483 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688859 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jun 02 01:41:22.691483 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688870 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jun 02 01:41:22.691483 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688875 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jun 02 01:41:22.691483 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688880 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jun 02 01:41:22.691483 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688885 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jun 02 01:41:22.691483 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688890 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jun 02 01:41:22.691483 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688895 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Jun 02 01:41:22.691483 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688899 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jun 02 01:41:22.691483 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688903 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jun 02 01:41:22.691483 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688907 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jun 02 01:41:22.691483 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688913 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jun 02 01:41:22.691483 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688917 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jun 02 01:41:22.691483 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688921 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jun 02 01:41:22.691483 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688926 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jun 02 01:41:22.691483 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688930 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Jun 02 01:41:22.691483 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688934 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jun 02 01:41:22.691483 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688940 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jun 02 01:41:22.691483 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688947 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jun 02 01:41:22.691483 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688951 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jun 02 01:41:22.692318 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688956 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jun 02 01:41:22.692318 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688960 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jun 02 01:41:22.692318 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688965 2571 feature_gate.go:328] unrecognized feature gate: Example Jun 02 01:41:22.692318 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688969 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS Jun 02 01:41:22.692318 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688973 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jun 02 01:41:22.692318 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688977 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jun 02 01:41:22.692318 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688982 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jun 02 01:41:22.692318 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688986 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jun 02 01:41:22.692318 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688992 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jun 02 01:41:22.692318 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.688997 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jun 02 01:41:22.692318 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.689003 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jun 02 01:41:22.692318 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.689008 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jun 02 01:41:22.692318 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.689012 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations Jun 02 01:41:22.692318 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.689016 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jun 02 01:41:22.692318 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.689020 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jun 02 01:41:22.692318 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.689024 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jun 02 01:41:22.692318 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.689029 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jun 02 01:41:22.692318 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.689033 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jun 02 01:41:22.692318 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.689037 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jun 02 01:41:22.692804 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.689041 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jun 02 01:41:22.692804 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.689046 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jun 02 01:41:22.692804 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.689050 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jun 02 01:41:22.692804 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.689054 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jun 02 01:41:22.692804 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.689059 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jun 02 01:41:22.692804 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.689064 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jun 02 01:41:22.692804 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.689068 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jun 02 01:41:22.692804 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.689073 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Jun 02 01:41:22.692804 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.689077 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jun 02 01:41:22.692804 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.689081 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jun 02 01:41:22.692804 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.689085 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jun 02 01:41:22.692804 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.689090 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jun 02 01:41:22.692804 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.689094 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jun 02 01:41:22.692804 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.689098 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jun 02 01:41:22.692804 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.689102 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jun 02 01:41:22.692804 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.689106 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jun 02 01:41:22.692804 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.689110 2571 feature_gate.go:328] unrecognized feature gate: Example2 Jun 02 01:41:22.692804 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.689114 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jun 02 01:41:22.692804 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.689118 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jun 02 01:41:22.692804 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.689122 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Jun 02 01:41:22.693432 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.689146 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jun 02 01:41:22.693432 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.689152 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jun 02 01:41:22.693432 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.689156 2571 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController Jun 02 01:41:22.693432 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.689161 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jun 02 01:41:22.693432 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.689165 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jun 02 01:41:22.693432 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.689169 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jun 02 01:41:22.693432 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.689173 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI Jun 02 01:41:22.693432 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.689177 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jun 02 01:41:22.693432 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.689182 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jun 02 01:41:22.693432 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.689185 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jun 02 01:41:22.693432 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.689190 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jun 02 01:41:22.693432 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.689194 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jun 02 01:41:22.693432 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.689198 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jun 02 01:41:22.693432 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.689202 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jun 02 01:41:22.693432 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.689206 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jun 02 01:41:22.693432 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.689211 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jun 02 01:41:22.693432 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.689215 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jun 02 01:41:22.693432 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.689219 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jun 02 01:41:22.693432 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.689224 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jun 02 01:41:22.693895 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.689228 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jun 02 01:41:22.693895 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.689232 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jun 02 01:41:22.693895 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.689236 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Jun 02 01:41:22.693895 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.689240 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jun 02 01:41:22.693895 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.689244 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jun 02 01:41:22.693895 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.689249 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jun 02 01:41:22.693895 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.689253 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jun 02 01:41:22.693895 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.689257 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jun 02 01:41:22.693895 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:22.689261 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jun 02 01:41:22.693895 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.689269 2571 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]} Jun 02 01:41:22.693895 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.690149 2571 server.go:962] "Client rotation is on, will bootstrap in background" Jun 02 01:41:22.693895 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.692906 2571 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jun 02 01:41:22.694356 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.693967 2571 server.go:1019] "Starting client certificate rotation" Jun 02 01:41:22.694356 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.694063 2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jun 02 01:41:22.694756 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.694744 2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jun 02 01:41:22.720402 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.720374 2571 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jun 02 01:41:22.722515 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.722495 2571 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jun 02 01:41:22.744655 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.744638 2571 log.go:25] "Validated CRI v1 runtime API" Jun 02 01:41:22.754283 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.754264 2571 log.go:25] "Validated CRI v1 image API" Jun 02 01:41:22.755502 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.755486 2571 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jun 02 01:41:22.758088 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.758069 2571 fs.go:135] Filesystem UUIDs: map[3b572faa-df8c-4f89-b2ab-b4b722f08974:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 bff885d5-a046-45e1-b5f0-332ef7b3143f:/dev/nvme0n1p4] Jun 02 01:41:22.758178 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.758087 2571 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jun 02 01:41:22.761471 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.761452 2571 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jun 02 01:41:22.762762 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.762654 2571 manager.go:217] Machine: {Timestamp:2026-06-02 01:41:22.762116648 +0000 UTC m=+0.407923301 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099881 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21b1b2a2a58ff24e5bc07322654f89 SystemUUID:ec21b1b2-a2a5-8ff2-4e5b-c07322654f89 BootID:da19677c-65da-4f31-8a9b-fcb79ab669b1 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:6b:2b:c5:5c:0f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:6b:2b:c5:5c:0f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:62:24:4d:3c:e0:18 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jun 02 01:41:22.762762 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.762758 2571 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jun 02 01:41:22.762902 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.762843 2571 manager.go:233] Version: {KernelVersion:5.14.0-570.117.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260521-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jun 02 01:41:22.765520 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.765495 2571 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jun 02 01:41:22.765689 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.765523 2571 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-117.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jun 02 01:41:22.766308 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.766296 2571 topology_manager.go:138] "Creating topology manager with none policy" Jun 02 01:41:22.766360 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.766313 2571 container_manager_linux.go:306] "Creating device plugin manager" Jun 02 01:41:22.766360 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.766324 2571 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jun 02 01:41:22.767810 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.767797 2571 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jun 02 01:41:22.769191 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.769179 2571 state_mem.go:36] "Initialized new in-memory state store" Jun 02 01:41:22.769497 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.769485 2571 server.go:1267] "Using root directory" path="/var/lib/kubelet" Jun 02 01:41:22.772162 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.772145 2571 kubelet.go:491] "Attempting to sync node with API server" Jun 02 01:41:22.772235 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.772168 2571 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests" Jun 02 01:41:22.772235 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.772215 2571 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jun 02 01:41:22.772235 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.772231 2571 kubelet.go:397] "Adding apiserver pod source" Jun 02 01:41:22.772358 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.772244 2571 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jun 02 01:41:22.773603 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.773587 2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jun 02 01:41:22.773685 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.773613 2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jun 02 01:41:22.776562 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.776548 2571 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.12-3.rhaos4.20.git64c6a00.el9" apiVersion="v1" Jun 02 01:41:22.778477 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.778463 2571 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jun 02 01:41:22.780384 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.780370 2571 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jun 02 01:41:22.780441 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.780389 2571 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jun 02 01:41:22.780441 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.780396 2571 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jun 02 01:41:22.780441 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.780403 2571 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jun 02 01:41:22.780441 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.780415 2571 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jun 02 01:41:22.780441 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.780425 2571 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jun 02 01:41:22.780441 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.780432 2571 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jun 02 01:41:22.780441 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.780438 2571 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jun 02 01:41:22.780629 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.780445 2571 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jun 02 01:41:22.780629 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.780452 2571 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jun 02 01:41:22.780629 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.780463 2571 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jun 02 01:41:22.780629 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.780473 2571 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jun 02 01:41:22.781250 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.781234 2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cdm2d" Jun 02 01:41:22.781623 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.781610 2571 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jun 02 01:41:22.781676 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.781626 2571 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image" Jun 02 01:41:22.783173 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:22.783149 2571 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jun 02 01:41:22.783173 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:22.783155 2571 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-117.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jun 02 01:41:22.784620 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.784607 2571 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-117.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jun 02 01:41:22.785189 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.785175 2571 watchdog_linux.go:99] "Systemd watchdog is not enabled" Jun 02 01:41:22.785228 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.785213 2571 server.go:1295] "Started kubelet" Jun 02 01:41:22.785386 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.785334 2571 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jun 02 01:41:22.785486 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.785399 2571 server_v1.go:47] "podresources" method="list" useActivePods=true Jun 02 01:41:22.786159 ip-10-0-135-117 systemd[1]: Started Kubernetes Kubelet. Jun 02 01:41:22.786391 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.786288 2571 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jun 02 01:41:22.786676 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.786628 2571 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jun 02 01:41:22.788563 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.788547 2571 server.go:317] "Adding debug handlers to kubelet server" Jun 02 01:41:22.788746 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.788730 2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cdm2d" Jun 02 01:41:22.791814 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.791796 2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jun 02 01:41:22.792221 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.792204 2571 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jun 02 01:41:22.792436 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.792352 2571 volume_manager.go:295] "The desired_state_of_world populator starts" Jun 02 01:41:22.792436 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.792366 2571 volume_manager.go:297] "Starting Kubelet Volume Manager" Jun 02 01:41:22.795008 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.794562 2571 desired_state_of_world_populator.go:150] "Desired state populator starts to run" Jun 02 01:41:22.795008 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.794677 2571 reconstruct.go:97] "Volume reconstruction finished" Jun 02 01:41:22.795008 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.794686 2571 reconciler.go:26] "Reconciler: start to sync state" Jun 02 01:41:22.796245 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:22.796213 2571 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-117.ec2.internal\" not found" Jun 02 01:41:22.796706 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.796690 2571 factory.go:55] Registering systemd factory Jun 02 01:41:22.796827 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.796815 2571 factory.go:223] Registration of the systemd container factory successfully Jun 02 01:41:22.797711 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.797695 2571 factory.go:153] Registering CRI-O factory Jun 02 01:41:22.797879 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.797858 2571 factory.go:223] Registration of the crio container factory successfully Jun 02 01:41:22.797979 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.797924 2571 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jun 02 01:41:22.797979 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.797953 2571 factory.go:103] Registering Raw factory Jun 02 01:41:22.797979 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.797970 2571 manager.go:1196] Started watching for new ooms in manager Jun 02 01:41:22.798883 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.798866 2571 manager.go:319] Starting recovery of all containers Jun 02 01:41:22.799572 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:22.799539 2571 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jun 02 01:41:22.804819 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.804657 2571 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jun 02 01:41:22.807472 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:22.807449 2571 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-135-117.ec2.internal\" not found" node="ip-10-0-135-117.ec2.internal" Jun 02 01:41:22.809396 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.809381 2571 manager.go:324] Recovery completed Jun 02 01:41:22.813309 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.813297 2571 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" Jun 02 01:41:22.816750 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.816734 2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-117.ec2.internal" event="NodeHasSufficientMemory" Jun 02 01:41:22.816820 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.816762 2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-117.ec2.internal" event="NodeHasNoDiskPressure" Jun 02 01:41:22.816820 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.816772 2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-117.ec2.internal" event="NodeHasSufficientPID" Jun 02 01:41:22.817288 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.817274 2571 cpu_manager.go:222] "Starting CPU manager" policy="none" Jun 02 01:41:22.817288 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.817285 2571 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jun 02 01:41:22.817393 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.817300 2571 state_mem.go:36] "Initialized new in-memory state store" Jun 02 01:41:22.819645 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.819634 2571 policy_none.go:49] "None policy: Start" Jun 02 01:41:22.819687 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.819649 2571 memory_manager.go:186] "Starting memorymanager" policy="None" Jun 02 01:41:22.819687 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.819658 2571 state_mem.go:35] "Initializing new in-memory state store" Jun 02 01:41:22.863037 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.860417 2571 manager.go:341] "Starting Device Plugin manager" Jun 02 01:41:22.863037 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:22.860444 2571 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jun 02 01:41:22.863037 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.860453 2571 server.go:85] "Starting device plugin registration server" Jun 02 01:41:22.863037 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.860704 2571 eviction_manager.go:189] "Eviction manager: starting control loop" Jun 02 01:41:22.863037 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.860718 2571 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jun 02 01:41:22.863037 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.860813 2571 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jun 02 01:41:22.863037 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.860885 2571 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jun 02 01:41:22.863037 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.860894 2571 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jun 02 01:41:22.863037 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:22.861316 2571 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jun 02 01:41:22.863037 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:22.861353 2571 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-117.ec2.internal\" not found" Jun 02 01:41:22.921998 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.921924 2571 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4" Jun 02 01:41:22.923023 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.923010 2571 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6" Jun 02 01:41:22.923110 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.923034 2571 status_manager.go:230] "Starting to sync pod status with apiserver" Jun 02 01:41:22.923110 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.923054 2571 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jun 02 01:41:22.923110 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.923060 2571 kubelet.go:2451] "Starting kubelet main sync loop" Jun 02 01:41:22.923110 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:22.923093 2571 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jun 02 01:41:22.927673 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.927658 2571 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jun 02 01:41:22.961078 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.961062 2571 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" Jun 02 01:41:22.962086 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.962069 2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-117.ec2.internal" event="NodeHasSufficientMemory" Jun 02 01:41:22.962171 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.962102 2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-117.ec2.internal" event="NodeHasNoDiskPressure" Jun 02 01:41:22.962171 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.962116 2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-117.ec2.internal" event="NodeHasSufficientPID" Jun 02 01:41:22.962171 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.962154 2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-117.ec2.internal" Jun 02 01:41:22.970835 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:22.970817 2571 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-117.ec2.internal" Jun 02 01:41:22.970898 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:22.970840 2571 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-117.ec2.internal\": node \"ip-10-0-135-117.ec2.internal\" not found" Jun 02 01:41:22.993461 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:22.993440 2571 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-117.ec2.internal\" not found" Jun 02 01:41:23.023699 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.023667 2571 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-117.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-117.ec2.internal"] Jun 02 01:41:23.023765 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.023755 2571 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" Jun 02 01:41:23.025469 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.025449 2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-117.ec2.internal" event="NodeHasSufficientMemory" Jun 02 01:41:23.025556 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.025478 2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-117.ec2.internal" event="NodeHasNoDiskPressure" Jun 02 01:41:23.025556 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.025490 2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-117.ec2.internal" event="NodeHasSufficientPID" Jun 02 01:41:23.026728 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.026716 2571 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" Jun 02 01:41:23.026916 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.026903 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-117.ec2.internal" Jun 02 01:41:23.026953 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.026933 2571 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" Jun 02 01:41:23.027459 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.027437 2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-117.ec2.internal" event="NodeHasSufficientMemory" Jun 02 01:41:23.027546 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.027466 2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-117.ec2.internal" event="NodeHasNoDiskPressure" Jun 02 01:41:23.027546 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.027477 2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-117.ec2.internal" event="NodeHasSufficientPID" Jun 02 01:41:23.027546 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.027438 2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-117.ec2.internal" event="NodeHasSufficientMemory" Jun 02 01:41:23.027655 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.027547 2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-117.ec2.internal" event="NodeHasNoDiskPressure" Jun 02 01:41:23.027655 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.027563 2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-117.ec2.internal" event="NodeHasSufficientPID" Jun 02 01:41:23.028582 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.028570 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-117.ec2.internal" Jun 02 01:41:23.028634 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.028598 2571 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" Jun 02 01:41:23.029212 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.029196 2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-117.ec2.internal" event="NodeHasSufficientMemory" Jun 02 01:41:23.029260 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.029228 2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-117.ec2.internal" event="NodeHasNoDiskPressure" Jun 02 01:41:23.029260 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.029254 2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-117.ec2.internal" event="NodeHasSufficientPID" Jun 02 01:41:23.054705 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:23.054682 2571 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-117.ec2.internal\" not found" node="ip-10-0-135-117.ec2.internal" Jun 02 01:41:23.058917 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:23.058902 2571 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-117.ec2.internal\" not found" node="ip-10-0-135-117.ec2.internal" Jun 02 01:41:23.094346 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:23.094323 2571 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-117.ec2.internal\" not found" Jun 02 01:41:23.096214 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.096199 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ee07ddb5f970db1345c2560076c9a2b8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-117.ec2.internal\" (UID: \"ee07ddb5f970db1345c2560076c9a2b8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-117.ec2.internal" Jun 02 01:41:23.096278 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.096222 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee07ddb5f970db1345c2560076c9a2b8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-117.ec2.internal\" (UID: \"ee07ddb5f970db1345c2560076c9a2b8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-117.ec2.internal" Jun 02 01:41:23.096627 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.096611 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dddc001fa74725ad60cc8708aa217f2d-config\") pod \"kube-apiserver-proxy-ip-10-0-135-117.ec2.internal\" (UID: \"dddc001fa74725ad60cc8708aa217f2d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-117.ec2.internal" Jun 02 01:41:23.194653 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:23.194577 2571 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-117.ec2.internal\" not found" Jun 02 01:41:23.196762 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.196741 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ee07ddb5f970db1345c2560076c9a2b8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-117.ec2.internal\" (UID: \"ee07ddb5f970db1345c2560076c9a2b8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-117.ec2.internal" Jun 02 01:41:23.196821 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.196772 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee07ddb5f970db1345c2560076c9a2b8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-117.ec2.internal\" (UID: \"ee07ddb5f970db1345c2560076c9a2b8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-117.ec2.internal" Jun 02 01:41:23.196821 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.196789 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dddc001fa74725ad60cc8708aa217f2d-config\") pod \"kube-apiserver-proxy-ip-10-0-135-117.ec2.internal\" (UID: \"dddc001fa74725ad60cc8708aa217f2d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-117.ec2.internal" Jun 02 01:41:23.196913 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.196839 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dddc001fa74725ad60cc8708aa217f2d-config\") pod \"kube-apiserver-proxy-ip-10-0-135-117.ec2.internal\" (UID: \"dddc001fa74725ad60cc8708aa217f2d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-117.ec2.internal" Jun 02 01:41:23.196913 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.196853 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee07ddb5f970db1345c2560076c9a2b8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-117.ec2.internal\" (UID: \"ee07ddb5f970db1345c2560076c9a2b8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-117.ec2.internal" Jun 02 01:41:23.196913 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.196861 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ee07ddb5f970db1345c2560076c9a2b8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-117.ec2.internal\" (UID: \"ee07ddb5f970db1345c2560076c9a2b8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-117.ec2.internal" Jun 02 01:41:23.295396 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:23.295367 2571 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-117.ec2.internal\" not found" Jun 02 01:41:23.357697 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.357674 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-117.ec2.internal" Jun 02 01:41:23.362307 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.362287 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-117.ec2.internal" Jun 02 01:41:23.396265 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:23.396232 2571 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-117.ec2.internal\" not found" Jun 02 01:41:23.496877 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:23.496783 2571 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-117.ec2.internal\" not found" Jun 02 01:41:23.593037 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.593006 2571 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jun 02 01:41:23.693117 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.693085 2571 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-117.ec2.internal" Jun 02 01:41:23.694226 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.694193 2571 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jun 02 01:41:23.694352 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:23.694318 2571 kubelet.go:3342] "Failed creating a mirror pod" err="Post \"https://a3aa42e41384546e281191cb33b463a4-3dd1856a46811cf0.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/openshift-machine-config-operator/pods\": write tcp 10.0.135.117:56080->52.206.114.106:6443: use of closed network connection" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-117.ec2.internal" Jun 02 01:41:23.694352 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.694329 2571 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection force closed via ClientConn.Close\") has prevented the request from succeeding" Jun 02 01:41:23.694352 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.694330 2571 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection force closed via ClientConn.Close\") has prevented the request from succeeding" Jun 02 01:41:23.694453 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.694330 2571 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection force closed via ClientConn.Close\") has prevented the request from succeeding" Jun 02 01:41:23.694453 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.694339 2571 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-117.ec2.internal" Jun 02 01:41:23.708975 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.708950 2571 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jun 02 01:41:23.712043 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.712027 2571 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jun 02 01:41:23.773280 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.773251 2571 apiserver.go:52] "Watching apiserver" Jun 02 01:41:23.779567 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.779543 2571 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jun 02 01:41:23.781557 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.781527 2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-thlx8","openshift-image-registry/node-ca-6rzvj","openshift-multus/multus-additional-cni-plugins-tglfq","openshift-ovn-kubernetes/ovnkube-node-fvdb7","kube-system/kube-apiserver-proxy-ip-10-0-135-117.ec2.internal","openshift-dns/node-resolver-nm89v","openshift-multus/multus-7drhb","openshift-multus/network-metrics-daemon-smqwh","openshift-network-diagnostics/network-check-target-gt4w8","openshift-network-operator/iptables-alerter-6dnbd","kube-system/konnectivity-agent-bmd65","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zf8v"] Jun 02 01:41:23.783091 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.783067 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-thlx8" Jun 02 01:41:23.784227 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.784202 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zf8v" Jun 02 01:41:23.785217 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.785197 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6rzvj" Jun 02 01:41:23.785583 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.785564 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jun 02 01:41:23.785667 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.785568 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jun 02 01:41:23.785667 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.785608 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-wd4bz\"" Jun 02 01:41:23.786399 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.786381 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jun 02 01:41:23.786539 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.786489 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tglfq" Jun 02 01:41:23.787260 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.786621 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-vcwnn\"" Jun 02 01:41:23.787260 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.786649 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jun 02 01:41:23.787260 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.786694 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jun 02 01:41:23.787492 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.787474 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jun 02 01:41:23.787492 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.787484 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-qlxkf\"" Jun 02 01:41:23.788083 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.787634 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jun 02 01:41:23.788083 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.787771 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jun 02 01:41:23.788083 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.787925 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" Jun 02 01:41:23.788577 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.788559 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jun 02 01:41:23.788697 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.788683 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jun 02 01:41:23.788976 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.788958 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jun 02 01:41:23.789076 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.789009 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jun 02 01:41:23.789155 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.789106 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nm89v" Jun 02 01:41:23.789155 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.789143 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-gwhgw\"" Jun 02 01:41:23.789265 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.789167 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jun 02 01:41:23.790362 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.790346 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jun 02 01:41:23.790362 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.790356 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jun 02 01:41:23.790459 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.790439 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jun 02 01:41:23.790505 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.790458 2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-06-01 01:36:22 +0000 UTC" deadline="2027-11-16 23:36:41.35013691 +0000 UTC" Jun 02 01:41:23.790539 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.790507 2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12789h55m17.559633909s" Jun 02 01:41:23.790539 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.790527 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jun 02 01:41:23.790596 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.790539 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-tr5xm\"" Jun 02 01:41:23.790596 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.790442 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7drhb" Jun 02 01:41:23.791984 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.791099 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jun 02 01:41:23.791984 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.791649 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jun 02 01:41:23.791984 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.791732 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-p4mwz\"" Jun 02 01:41:23.791984 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.791747 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jun 02 01:41:23.791984 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.791822 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jun 02 01:41:23.792338 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.792060 2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jun 02 01:41:23.793591 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.793196 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jun 02 01:41:23.793591 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.793208 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-29sxf\"" Jun 02 01:41:23.794065 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.794048 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-smqwh" Jun 02 01:41:23.794207 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:23.794147 2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-smqwh" podUID="f3542744-21d2-4162-a2f9-4a474d18dc19" Jun 02 01:41:23.794207 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.794164 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gt4w8" Jun 02 01:41:23.794357 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:23.794213 2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gt4w8" podUID="7db70fbd-3197-47ed-a00b-d6b4fbb4c8a8" Jun 02 01:41:23.795222 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.795208 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6dnbd" Jun 02 01:41:23.796380 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.796361 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bmd65" Jun 02 01:41:23.797214 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.797200 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jun 02 01:41:23.797289 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.797274 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jun 02 01:41:23.797457 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.797439 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qxtc8\"" Jun 02 01:41:23.797534 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.797503 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jun 02 01:41:23.798415 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.798401 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jun 02 01:41:23.798508 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.798486 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-9pqwn\"" Jun 02 01:41:23.798607 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.798593 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jun 02 01:41:23.801677 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.801656 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aa0d7d15-d800-4698-a39f-763402e48110-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fvdb7\" (UID: \"aa0d7d15-d800-4698-a39f-763402e48110\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" Jun 02 01:41:23.801758 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.801684 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/aa0d7d15-d800-4698-a39f-763402e48110-host-kubelet\") pod \"ovnkube-node-fvdb7\" (UID: \"aa0d7d15-d800-4698-a39f-763402e48110\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" Jun 02 01:41:23.801758 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.801701 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aa0d7d15-d800-4698-a39f-763402e48110-host-run-netns\") pod \"ovnkube-node-fvdb7\" (UID: \"aa0d7d15-d800-4698-a39f-763402e48110\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" Jun 02 01:41:23.801758 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.801720 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aa0d7d15-d800-4698-a39f-763402e48110-etc-openvswitch\") pod \"ovnkube-node-fvdb7\" (UID: \"aa0d7d15-d800-4698-a39f-763402e48110\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" Jun 02 01:41:23.801758 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.801768 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aa0d7d15-d800-4698-a39f-763402e48110-run-openvswitch\") pod \"ovnkube-node-fvdb7\" (UID: \"aa0d7d15-d800-4698-a39f-763402e48110\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" Jun 02 01:41:23.801967 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.801793 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aa0d7d15-d800-4698-a39f-763402e48110-host-run-ovn-kubernetes\") pod \"ovnkube-node-fvdb7\" (UID: \"aa0d7d15-d800-4698-a39f-763402e48110\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" Jun 02 01:41:23.801967 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.801826 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5cffe19e-fcbd-46bb-9db3-2617a77f95f6-socket-dir\") pod \"aws-ebs-csi-driver-node-7zf8v\" (UID: \"5cffe19e-fcbd-46bb-9db3-2617a77f95f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zf8v" Jun 02 01:41:23.801967 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.801859 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ba65c8da-80cd-4fda-9746-dd851a3a06be-multus-conf-dir\") pod \"multus-7drhb\" (UID: \"ba65c8da-80cd-4fda-9746-dd851a3a06be\") " pod="openshift-multus/multus-7drhb" Jun 02 01:41:23.801967 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.801875 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ae3e1499-5744-4270-b94d-f292d30bd1b2-cni-binary-copy\") pod \"multus-additional-cni-plugins-tglfq\" (UID: \"ae3e1499-5744-4270-b94d-f292d30bd1b2\") " pod="openshift-multus/multus-additional-cni-plugins-tglfq" Jun 02 01:41:23.801967 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.801893 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/aa0d7d15-d800-4698-a39f-763402e48110-host-cni-netd\") pod \"ovnkube-node-fvdb7\" (UID: \"aa0d7d15-d800-4698-a39f-763402e48110\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" Jun 02 01:41:23.801967 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.801925 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ba65c8da-80cd-4fda-9746-dd851a3a06be-multus-daemon-config\") pod \"multus-7drhb\" (UID: \"ba65c8da-80cd-4fda-9746-dd851a3a06be\") " pod="openshift-multus/multus-7drhb" Jun 02 01:41:23.802227 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.801979 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b78335d-1f32-4d95-adda-efac2d74e4e1-var-lib-kubelet\") pod \"tuned-thlx8\" (UID: \"4b78335d-1f32-4d95-adda-efac2d74e4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-thlx8" Jun 02 01:41:23.802227 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.802012 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ae3e1499-5744-4270-b94d-f292d30bd1b2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tglfq\" (UID: \"ae3e1499-5744-4270-b94d-f292d30bd1b2\") " pod="openshift-multus/multus-additional-cni-plugins-tglfq" Jun 02 01:41:23.802227 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.802043 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ba65c8da-80cd-4fda-9746-dd851a3a06be-multus-cni-dir\") pod \"multus-7drhb\" (UID: \"ba65c8da-80cd-4fda-9746-dd851a3a06be\") " pod="openshift-multus/multus-7drhb" Jun 02 01:41:23.802227 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.802059 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ba65c8da-80cd-4fda-9746-dd851a3a06be-multus-socket-dir-parent\") pod \"multus-7drhb\" (UID: \"ba65c8da-80cd-4fda-9746-dd851a3a06be\") " pod="openshift-multus/multus-7drhb" Jun 02 01:41:23.802227 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.802077 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ba65c8da-80cd-4fda-9746-dd851a3a06be-host-run-k8s-cni-cncf-io\") pod \"multus-7drhb\" (UID: \"ba65c8da-80cd-4fda-9746-dd851a3a06be\") " pod="openshift-multus/multus-7drhb" Jun 02 01:41:23.802227 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.802104 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ba65c8da-80cd-4fda-9746-dd851a3a06be-host-var-lib-cni-bin\") pod \"multus-7drhb\" (UID: \"ba65c8da-80cd-4fda-9746-dd851a3a06be\") " pod="openshift-multus/multus-7drhb" Jun 02 01:41:23.802227 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.802120 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4b78335d-1f32-4d95-adda-efac2d74e4e1-etc-systemd\") pod \"tuned-thlx8\" (UID: \"4b78335d-1f32-4d95-adda-efac2d74e4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-thlx8" Jun 02 01:41:23.802227 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.802152 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4b78335d-1f32-4d95-adda-efac2d74e4e1-etc-tuned\") pod \"tuned-thlx8\" (UID: \"4b78335d-1f32-4d95-adda-efac2d74e4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-thlx8" Jun 02 01:41:23.802227 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.802171 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c45e264-330c-4438-bdc1-817a0a8b1b77-tmp-dir\") pod \"node-resolver-nm89v\" (UID: \"5c45e264-330c-4438-bdc1-817a0a8b1b77\") " pod="openshift-dns/node-resolver-nm89v" Jun 02 01:41:23.802227 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.802185 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5cffe19e-fcbd-46bb-9db3-2617a77f95f6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7zf8v\" (UID: \"5cffe19e-fcbd-46bb-9db3-2617a77f95f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zf8v" Jun 02 01:41:23.802227 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.802198 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5cffe19e-fcbd-46bb-9db3-2617a77f95f6-sys-fs\") pod \"aws-ebs-csi-driver-node-7zf8v\" (UID: \"5cffe19e-fcbd-46bb-9db3-2617a77f95f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zf8v" Jun 02 01:41:23.802227 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.802217 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4b78335d-1f32-4d95-adda-efac2d74e4e1-etc-sysctl-conf\") pod \"tuned-thlx8\" (UID: \"4b78335d-1f32-4d95-adda-efac2d74e4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-thlx8" Jun 02 01:41:23.802681 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.802237 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wccb2\" (UniqueName: \"kubernetes.io/projected/ae3e1499-5744-4270-b94d-f292d30bd1b2-kube-api-access-wccb2\") pod \"multus-additional-cni-plugins-tglfq\" (UID: \"ae3e1499-5744-4270-b94d-f292d30bd1b2\") " pod="openshift-multus/multus-additional-cni-plugins-tglfq" Jun 02 01:41:23.802681 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.802251 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ba65c8da-80cd-4fda-9746-dd851a3a06be-host-run-multus-certs\") pod \"multus-7drhb\" (UID: \"ba65c8da-80cd-4fda-9746-dd851a3a06be\") " pod="openshift-multus/multus-7drhb" Jun 02 01:41:23.802681 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.802263 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4b78335d-1f32-4d95-adda-efac2d74e4e1-host\") pod \"tuned-thlx8\" (UID: \"4b78335d-1f32-4d95-adda-efac2d74e4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-thlx8" Jun 02 01:41:23.802681 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.802283 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/aa0d7d15-d800-4698-a39f-763402e48110-node-log\") pod \"ovnkube-node-fvdb7\" (UID: \"aa0d7d15-d800-4698-a39f-763402e48110\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" Jun 02 01:41:23.802681 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.802304 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/aa0d7d15-d800-4698-a39f-763402e48110-log-socket\") pod \"ovnkube-node-fvdb7\" (UID: \"aa0d7d15-d800-4698-a39f-763402e48110\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" Jun 02 01:41:23.802681 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.802327 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/aa0d7d15-d800-4698-a39f-763402e48110-ovnkube-config\") pod \"ovnkube-node-fvdb7\" (UID: \"aa0d7d15-d800-4698-a39f-763402e48110\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" Jun 02 01:41:23.802681 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.802344 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/aa0d7d15-d800-4698-a39f-763402e48110-ovnkube-script-lib\") pod \"ovnkube-node-fvdb7\" (UID: \"aa0d7d15-d800-4698-a39f-763402e48110\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" Jun 02 01:41:23.802681 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.802360 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5c45e264-330c-4438-bdc1-817a0a8b1b77-hosts-file\") pod \"node-resolver-nm89v\" (UID: \"5c45e264-330c-4438-bdc1-817a0a8b1b77\") " pod="openshift-dns/node-resolver-nm89v" Jun 02 01:41:23.802681 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.802374 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ba65c8da-80cd-4fda-9746-dd851a3a06be-host-var-lib-cni-multus\") pod \"multus-7drhb\" (UID: \"ba65c8da-80cd-4fda-9746-dd851a3a06be\") " pod="openshift-multus/multus-7drhb" Jun 02 01:41:23.802681 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.802387 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/aa0d7d15-d800-4698-a39f-763402e48110-systemd-units\") pod \"ovnkube-node-fvdb7\" (UID: \"aa0d7d15-d800-4698-a39f-763402e48110\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" Jun 02 01:41:23.802681 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.802400 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/aa0d7d15-d800-4698-a39f-763402e48110-run-ovn\") pod \"ovnkube-node-fvdb7\" (UID: \"aa0d7d15-d800-4698-a39f-763402e48110\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" Jun 02 01:41:23.802681 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.802420 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4b78335d-1f32-4d95-adda-efac2d74e4e1-etc-sysconfig\") pod \"tuned-thlx8\" (UID: \"4b78335d-1f32-4d95-adda-efac2d74e4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-thlx8" Jun 02 01:41:23.802681 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.802434 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4b78335d-1f32-4d95-adda-efac2d74e4e1-lib-modules\") pod \"tuned-thlx8\" (UID: \"4b78335d-1f32-4d95-adda-efac2d74e4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-thlx8" Jun 02 01:41:23.802681 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.802447 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/aa0d7d15-d800-4698-a39f-763402e48110-env-overrides\") pod \"ovnkube-node-fvdb7\" (UID: \"aa0d7d15-d800-4698-a39f-763402e48110\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" Jun 02 01:41:23.802681 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.802461 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ba65c8da-80cd-4fda-9746-dd851a3a06be-os-release\") pod \"multus-7drhb\" (UID: \"ba65c8da-80cd-4fda-9746-dd851a3a06be\") " pod="openshift-multus/multus-7drhb" Jun 02 01:41:23.802681 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.802475 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kbsh\" (UniqueName: \"kubernetes.io/projected/ba65c8da-80cd-4fda-9746-dd851a3a06be-kube-api-access-5kbsh\") pod \"multus-7drhb\" (UID: \"ba65c8da-80cd-4fda-9746-dd851a3a06be\") " pod="openshift-multus/multus-7drhb" Jun 02 01:41:23.802681 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.802489 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8f82072c-4fa6-41c2-91ce-6c7cd16a0840-iptables-alerter-script\") pod \"iptables-alerter-6dnbd\" (UID: \"8f82072c-4fa6-41c2-91ce-6c7cd16a0840\") " pod="openshift-network-operator/iptables-alerter-6dnbd" Jun 02 01:41:23.803270 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.802516 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4b78335d-1f32-4d95-adda-efac2d74e4e1-run\") pod \"tuned-thlx8\" (UID: \"4b78335d-1f32-4d95-adda-efac2d74e4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-thlx8" Jun 02 01:41:23.803270 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.802539 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ae3e1499-5744-4270-b94d-f292d30bd1b2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tglfq\" (UID: \"ae3e1499-5744-4270-b94d-f292d30bd1b2\") " pod="openshift-multus/multus-additional-cni-plugins-tglfq" Jun 02 01:41:23.803270 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.802557 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/aa0d7d15-d800-4698-a39f-763402e48110-run-systemd\") pod \"ovnkube-node-fvdb7\" (UID: \"aa0d7d15-d800-4698-a39f-763402e48110\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" Jun 02 01:41:23.803270 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.802572 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aa0d7d15-d800-4698-a39f-763402e48110-var-lib-openvswitch\") pod \"ovnkube-node-fvdb7\" (UID: \"aa0d7d15-d800-4698-a39f-763402e48110\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" Jun 02 01:41:23.803270 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.802587 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5cffe19e-fcbd-46bb-9db3-2617a77f95f6-device-dir\") pod \"aws-ebs-csi-driver-node-7zf8v\" (UID: \"5cffe19e-fcbd-46bb-9db3-2617a77f95f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zf8v" Jun 02 01:41:23.803270 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.802607 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ba65c8da-80cd-4fda-9746-dd851a3a06be-host-run-netns\") pod \"multus-7drhb\" (UID: \"ba65c8da-80cd-4fda-9746-dd851a3a06be\") " pod="openshift-multus/multus-7drhb" Jun 02 01:41:23.803270 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.802620 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ba65c8da-80cd-4fda-9746-dd851a3a06be-hostroot\") pod \"multus-7drhb\" (UID: \"ba65c8da-80cd-4fda-9746-dd851a3a06be\") " pod="openshift-multus/multus-7drhb" Jun 02 01:41:23.803270 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.802634 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba65c8da-80cd-4fda-9746-dd851a3a06be-etc-kubernetes\") pod \"multus-7drhb\" (UID: \"ba65c8da-80cd-4fda-9746-dd851a3a06be\") " pod="openshift-multus/multus-7drhb" Jun 02 01:41:23.803270 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.802652 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1fc82c63-2678-486d-9393-c3ea7f27abce-host\") pod \"node-ca-6rzvj\" (UID: \"1fc82c63-2678-486d-9393-c3ea7f27abce\") " pod="openshift-image-registry/node-ca-6rzvj" Jun 02 01:41:23.803270 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.802667 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtw5j\" (UniqueName: \"kubernetes.io/projected/aa0d7d15-d800-4698-a39f-763402e48110-kube-api-access-xtw5j\") pod \"ovnkube-node-fvdb7\" (UID: \"aa0d7d15-d800-4698-a39f-763402e48110\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" Jun 02 01:41:23.803270 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.802681 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5cffe19e-fcbd-46bb-9db3-2617a77f95f6-etc-selinux\") pod \"aws-ebs-csi-driver-node-7zf8v\" (UID: \"5cffe19e-fcbd-46bb-9db3-2617a77f95f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zf8v" Jun 02 01:41:23.803270 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.802694 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4b78335d-1f32-4d95-adda-efac2d74e4e1-etc-modprobe-d\") pod \"tuned-thlx8\" (UID: \"4b78335d-1f32-4d95-adda-efac2d74e4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-thlx8" Jun 02 01:41:23.803270 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.802709 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1fc82c63-2678-486d-9393-c3ea7f27abce-serviceca\") pod \"node-ca-6rzvj\" (UID: \"1fc82c63-2678-486d-9393-c3ea7f27abce\") " pod="openshift-image-registry/node-ca-6rzvj" Jun 02 01:41:23.803270 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.802725 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ae3e1499-5744-4270-b94d-f292d30bd1b2-system-cni-dir\") pod \"multus-additional-cni-plugins-tglfq\" (UID: \"ae3e1499-5744-4270-b94d-f292d30bd1b2\") " pod="openshift-multus/multus-additional-cni-plugins-tglfq" Jun 02 01:41:23.803270 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.802739 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ba65c8da-80cd-4fda-9746-dd851a3a06be-cnibin\") pod \"multus-7drhb\" (UID: \"ba65c8da-80cd-4fda-9746-dd851a3a06be\") " pod="openshift-multus/multus-7drhb" Jun 02 01:41:23.803270 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.802754 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b78335d-1f32-4d95-adda-efac2d74e4e1-etc-kubernetes\") pod \"tuned-thlx8\" (UID: \"4b78335d-1f32-4d95-adda-efac2d74e4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-thlx8" Jun 02 01:41:23.803270 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.802774 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aa0d7d15-d800-4698-a39f-763402e48110-host-cni-bin\") pod \"ovnkube-node-fvdb7\" (UID: \"aa0d7d15-d800-4698-a39f-763402e48110\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" Jun 02 01:41:23.803847 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.802788 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ae3e1499-5744-4270-b94d-f292d30bd1b2-cnibin\") pod \"multus-additional-cni-plugins-tglfq\" (UID: \"ae3e1499-5744-4270-b94d-f292d30bd1b2\") " pod="openshift-multus/multus-additional-cni-plugins-tglfq" Jun 02 01:41:23.803847 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.802833 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/aa0d7d15-d800-4698-a39f-763402e48110-ovn-node-metrics-cert\") pod \"ovnkube-node-fvdb7\" (UID: \"aa0d7d15-d800-4698-a39f-763402e48110\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" Jun 02 01:41:23.803847 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.802858 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f7hb\" (UniqueName: \"kubernetes.io/projected/7db70fbd-3197-47ed-a00b-d6b4fbb4c8a8-kube-api-access-6f7hb\") pod \"network-check-target-gt4w8\" (UID: \"7db70fbd-3197-47ed-a00b-d6b4fbb4c8a8\") " pod="openshift-network-diagnostics/network-check-target-gt4w8" Jun 02 01:41:23.803847 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.802877 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh8f2\" (UniqueName: \"kubernetes.io/projected/5cffe19e-fcbd-46bb-9db3-2617a77f95f6-kube-api-access-qh8f2\") pod \"aws-ebs-csi-driver-node-7zf8v\" (UID: \"5cffe19e-fcbd-46bb-9db3-2617a77f95f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zf8v" Jun 02 01:41:23.803847 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.802898 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4b78335d-1f32-4d95-adda-efac2d74e4e1-etc-sysctl-d\") pod \"tuned-thlx8\" (UID: \"4b78335d-1f32-4d95-adda-efac2d74e4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-thlx8" Jun 02 01:41:23.803847 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.802924 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b78335d-1f32-4d95-adda-efac2d74e4e1-tmp\") pod \"tuned-thlx8\" (UID: \"4b78335d-1f32-4d95-adda-efac2d74e4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-thlx8" Jun 02 01:41:23.803847 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.802943 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvkvg\" (UniqueName: \"kubernetes.io/projected/4b78335d-1f32-4d95-adda-efac2d74e4e1-kube-api-access-xvkvg\") pod \"tuned-thlx8\" (UID: \"4b78335d-1f32-4d95-adda-efac2d74e4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-thlx8" Jun 02 01:41:23.803847 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.802961 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-467r4\" (UniqueName: \"kubernetes.io/projected/5c45e264-330c-4438-bdc1-817a0a8b1b77-kube-api-access-467r4\") pod \"node-resolver-nm89v\" (UID: \"5c45e264-330c-4438-bdc1-817a0a8b1b77\") " pod="openshift-dns/node-resolver-nm89v" Jun 02 01:41:23.803847 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.802982 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ba65c8da-80cd-4fda-9746-dd851a3a06be-system-cni-dir\") pod \"multus-7drhb\" (UID: \"ba65c8da-80cd-4fda-9746-dd851a3a06be\") " pod="openshift-multus/multus-7drhb" Jun 02 01:41:23.803847 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.802998 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba65c8da-80cd-4fda-9746-dd851a3a06be-host-var-lib-kubelet\") pod \"multus-7drhb\" (UID: \"ba65c8da-80cd-4fda-9746-dd851a3a06be\") " pod="openshift-multus/multus-7drhb" Jun 02 01:41:23.803847 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.803021 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8f82072c-4fa6-41c2-91ce-6c7cd16a0840-host-slash\") pod \"iptables-alerter-6dnbd\" (UID: \"8f82072c-4fa6-41c2-91ce-6c7cd16a0840\") " pod="openshift-network-operator/iptables-alerter-6dnbd" Jun 02 01:41:23.803847 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.803056 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qwx4\" (UniqueName: \"kubernetes.io/projected/1fc82c63-2678-486d-9393-c3ea7f27abce-kube-api-access-5qwx4\") pod \"node-ca-6rzvj\" (UID: \"1fc82c63-2678-486d-9393-c3ea7f27abce\") " pod="openshift-image-registry/node-ca-6rzvj" Jun 02 01:41:23.803847 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.803090 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aa0d7d15-d800-4698-a39f-763402e48110-host-slash\") pod \"ovnkube-node-fvdb7\" (UID: \"aa0d7d15-d800-4698-a39f-763402e48110\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" Jun 02 01:41:23.803847 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.803114 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3542744-21d2-4162-a2f9-4a474d18dc19-metrics-certs\") pod \"network-metrics-daemon-smqwh\" (UID: \"f3542744-21d2-4162-a2f9-4a474d18dc19\") " pod="openshift-multus/network-metrics-daemon-smqwh" Jun 02 01:41:23.803847 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.803154 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d9fc\" (UniqueName: \"kubernetes.io/projected/f3542744-21d2-4162-a2f9-4a474d18dc19-kube-api-access-5d9fc\") pod \"network-metrics-daemon-smqwh\" (UID: \"f3542744-21d2-4162-a2f9-4a474d18dc19\") " pod="openshift-multus/network-metrics-daemon-smqwh" Jun 02 01:41:23.803847 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.803172 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4bbk\" (UniqueName: \"kubernetes.io/projected/8f82072c-4fa6-41c2-91ce-6c7cd16a0840-kube-api-access-m4bbk\") pod \"iptables-alerter-6dnbd\" (UID: \"8f82072c-4fa6-41c2-91ce-6c7cd16a0840\") " pod="openshift-network-operator/iptables-alerter-6dnbd" Jun 02 01:41:23.804474 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.803185 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b78335d-1f32-4d95-adda-efac2d74e4e1-sys\") pod \"tuned-thlx8\" (UID: \"4b78335d-1f32-4d95-adda-efac2d74e4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-thlx8" Jun 02 01:41:23.804474 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.803204 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ae3e1499-5744-4270-b94d-f292d30bd1b2-os-release\") pod \"multus-additional-cni-plugins-tglfq\" (UID: \"ae3e1499-5744-4270-b94d-f292d30bd1b2\") " pod="openshift-multus/multus-additional-cni-plugins-tglfq" Jun 02 01:41:23.804474 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.803231 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ae3e1499-5744-4270-b94d-f292d30bd1b2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tglfq\" (UID: \"ae3e1499-5744-4270-b94d-f292d30bd1b2\") " pod="openshift-multus/multus-additional-cni-plugins-tglfq" Jun 02 01:41:23.804474 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.803245 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5cffe19e-fcbd-46bb-9db3-2617a77f95f6-registration-dir\") pod \"aws-ebs-csi-driver-node-7zf8v\" (UID: \"5cffe19e-fcbd-46bb-9db3-2617a77f95f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zf8v" Jun 02 01:41:23.804474 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.803264 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ba65c8da-80cd-4fda-9746-dd851a3a06be-cni-binary-copy\") pod \"multus-7drhb\" (UID: \"ba65c8da-80cd-4fda-9746-dd851a3a06be\") " pod="openshift-multus/multus-7drhb" Jun 02 01:41:23.806784 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.806770 2571 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jun 02 01:41:23.823582 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.823562 2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-s8lrp" Jun 02 01:41:23.830900 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.830881 2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-s8lrp" Jun 02 01:41:23.857458 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:23.857430 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddddc001fa74725ad60cc8708aa217f2d.slice/crio-9e4e2c22b00e1bb1a54bc1f897b720b20bb6e6a92284d6d75d431329472a4bfb WatchSource:0}: Error finding container 9e4e2c22b00e1bb1a54bc1f897b720b20bb6e6a92284d6d75d431329472a4bfb: Status 404 returned error can't find the container with id 9e4e2c22b00e1bb1a54bc1f897b720b20bb6e6a92284d6d75d431329472a4bfb Jun 02 01:41:23.857662 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:23.857640 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee07ddb5f970db1345c2560076c9a2b8.slice/crio-864eb063f0e598776ff1d4f09119033898db53ef5bbbd554343f6fd06f6d6c27 WatchSource:0}: Error finding container 864eb063f0e598776ff1d4f09119033898db53ef5bbbd554343f6fd06f6d6c27: Status 404 returned error can't find the container with id 864eb063f0e598776ff1d4f09119033898db53ef5bbbd554343f6fd06f6d6c27 Jun 02 01:41:23.862157 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.862106 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jun 02 01:41:23.895946 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.895930 2571 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world" Jun 02 01:41:23.903736 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.903714 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ba65c8da-80cd-4fda-9746-dd851a3a06be-host-run-netns\") pod \"multus-7drhb\" (UID: \"ba65c8da-80cd-4fda-9746-dd851a3a06be\") " pod="openshift-multus/multus-7drhb" Jun 02 01:41:23.903837 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.903741 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ba65c8da-80cd-4fda-9746-dd851a3a06be-hostroot\") pod \"multus-7drhb\" (UID: \"ba65c8da-80cd-4fda-9746-dd851a3a06be\") " pod="openshift-multus/multus-7drhb" Jun 02 01:41:23.903837 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.903758 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba65c8da-80cd-4fda-9746-dd851a3a06be-etc-kubernetes\") pod \"multus-7drhb\" (UID: \"ba65c8da-80cd-4fda-9746-dd851a3a06be\") " pod="openshift-multus/multus-7drhb" Jun 02 01:41:23.903837 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.903774 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1fc82c63-2678-486d-9393-c3ea7f27abce-host\") pod \"node-ca-6rzvj\" (UID: \"1fc82c63-2678-486d-9393-c3ea7f27abce\") " pod="openshift-image-registry/node-ca-6rzvj" Jun 02 01:41:23.903837 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.903790 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xtw5j\" (UniqueName: \"kubernetes.io/projected/aa0d7d15-d800-4698-a39f-763402e48110-kube-api-access-xtw5j\") pod \"ovnkube-node-fvdb7\" (UID: \"aa0d7d15-d800-4698-a39f-763402e48110\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" Jun 02 01:41:23.903837 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.903805 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5cffe19e-fcbd-46bb-9db3-2617a77f95f6-etc-selinux\") pod \"aws-ebs-csi-driver-node-7zf8v\" (UID: \"5cffe19e-fcbd-46bb-9db3-2617a77f95f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zf8v" Jun 02 01:41:23.903837 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.903815 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ba65c8da-80cd-4fda-9746-dd851a3a06be-host-run-netns\") pod \"multus-7drhb\" (UID: \"ba65c8da-80cd-4fda-9746-dd851a3a06be\") " pod="openshift-multus/multus-7drhb" Jun 02 01:41:23.903837 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.903825 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4b78335d-1f32-4d95-adda-efac2d74e4e1-etc-modprobe-d\") pod \"tuned-thlx8\" (UID: \"4b78335d-1f32-4d95-adda-efac2d74e4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-thlx8" Jun 02 01:41:23.903837 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.903816 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ba65c8da-80cd-4fda-9746-dd851a3a06be-hostroot\") pod \"multus-7drhb\" (UID: \"ba65c8da-80cd-4fda-9746-dd851a3a06be\") " pod="openshift-multus/multus-7drhb" Jun 02 01:41:23.904102 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.903839 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba65c8da-80cd-4fda-9746-dd851a3a06be-etc-kubernetes\") pod \"multus-7drhb\" (UID: \"ba65c8da-80cd-4fda-9746-dd851a3a06be\") " pod="openshift-multus/multus-7drhb" Jun 02 01:41:23.904102 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.903851 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1fc82c63-2678-486d-9393-c3ea7f27abce-host\") pod \"node-ca-6rzvj\" (UID: \"1fc82c63-2678-486d-9393-c3ea7f27abce\") " pod="openshift-image-registry/node-ca-6rzvj" Jun 02 01:41:23.904102 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.903927 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5cffe19e-fcbd-46bb-9db3-2617a77f95f6-etc-selinux\") pod \"aws-ebs-csi-driver-node-7zf8v\" (UID: \"5cffe19e-fcbd-46bb-9db3-2617a77f95f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zf8v" Jun 02 01:41:23.904102 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.903947 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4b78335d-1f32-4d95-adda-efac2d74e4e1-etc-modprobe-d\") pod \"tuned-thlx8\" (UID: \"4b78335d-1f32-4d95-adda-efac2d74e4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-thlx8" Jun 02 01:41:23.904102 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.903952 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1fc82c63-2678-486d-9393-c3ea7f27abce-serviceca\") pod \"node-ca-6rzvj\" (UID: \"1fc82c63-2678-486d-9393-c3ea7f27abce\") " pod="openshift-image-registry/node-ca-6rzvj" Jun 02 01:41:23.904102 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.903984 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ab3aef63-c4c0-4b01-b26b-53ca0e39e7bb-agent-certs\") pod \"konnectivity-agent-bmd65\" (UID: \"ab3aef63-c4c0-4b01-b26b-53ca0e39e7bb\") " pod="kube-system/konnectivity-agent-bmd65" Jun 02 01:41:23.904102 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.904012 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ae3e1499-5744-4270-b94d-f292d30bd1b2-system-cni-dir\") pod \"multus-additional-cni-plugins-tglfq\" (UID: \"ae3e1499-5744-4270-b94d-f292d30bd1b2\") " pod="openshift-multus/multus-additional-cni-plugins-tglfq" Jun 02 01:41:23.904102 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.904035 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ae3e1499-5744-4270-b94d-f292d30bd1b2-system-cni-dir\") pod \"multus-additional-cni-plugins-tglfq\" (UID: \"ae3e1499-5744-4270-b94d-f292d30bd1b2\") " pod="openshift-multus/multus-additional-cni-plugins-tglfq" Jun 02 01:41:23.904102 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.904036 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ba65c8da-80cd-4fda-9746-dd851a3a06be-cnibin\") pod \"multus-7drhb\" (UID: \"ba65c8da-80cd-4fda-9746-dd851a3a06be\") " pod="openshift-multus/multus-7drhb" Jun 02 01:41:23.904102 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.904064 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b78335d-1f32-4d95-adda-efac2d74e4e1-etc-kubernetes\") pod \"tuned-thlx8\" (UID: \"4b78335d-1f32-4d95-adda-efac2d74e4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-thlx8" Jun 02 01:41:23.904102 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.904080 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aa0d7d15-d800-4698-a39f-763402e48110-host-cni-bin\") pod \"ovnkube-node-fvdb7\" (UID: \"aa0d7d15-d800-4698-a39f-763402e48110\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" Jun 02 01:41:23.904102 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.904080 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ba65c8da-80cd-4fda-9746-dd851a3a06be-cnibin\") pod \"multus-7drhb\" (UID: \"ba65c8da-80cd-4fda-9746-dd851a3a06be\") " pod="openshift-multus/multus-7drhb" Jun 02 01:41:23.904102 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.904095 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ae3e1499-5744-4270-b94d-f292d30bd1b2-cnibin\") pod \"multus-additional-cni-plugins-tglfq\" (UID: \"ae3e1499-5744-4270-b94d-f292d30bd1b2\") " pod="openshift-multus/multus-additional-cni-plugins-tglfq" Jun 02 01:41:23.904633 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.904111 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/aa0d7d15-d800-4698-a39f-763402e48110-ovn-node-metrics-cert\") pod \"ovnkube-node-fvdb7\" (UID: \"aa0d7d15-d800-4698-a39f-763402e48110\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" Jun 02 01:41:23.904633 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.904147 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6f7hb\" (UniqueName: \"kubernetes.io/projected/7db70fbd-3197-47ed-a00b-d6b4fbb4c8a8-kube-api-access-6f7hb\") pod \"network-check-target-gt4w8\" (UID: \"7db70fbd-3197-47ed-a00b-d6b4fbb4c8a8\") " pod="openshift-network-diagnostics/network-check-target-gt4w8" Jun 02 01:41:23.904633 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.904150 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b78335d-1f32-4d95-adda-efac2d74e4e1-etc-kubernetes\") pod \"tuned-thlx8\" (UID: \"4b78335d-1f32-4d95-adda-efac2d74e4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-thlx8" Jun 02 01:41:23.904633 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.904150 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aa0d7d15-d800-4698-a39f-763402e48110-host-cni-bin\") pod \"ovnkube-node-fvdb7\" (UID: \"aa0d7d15-d800-4698-a39f-763402e48110\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" Jun 02 01:41:23.904633 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.904173 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qh8f2\" (UniqueName: \"kubernetes.io/projected/5cffe19e-fcbd-46bb-9db3-2617a77f95f6-kube-api-access-qh8f2\") pod \"aws-ebs-csi-driver-node-7zf8v\" (UID: \"5cffe19e-fcbd-46bb-9db3-2617a77f95f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zf8v" Jun 02 01:41:23.904633 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.904197 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4b78335d-1f32-4d95-adda-efac2d74e4e1-etc-sysctl-d\") pod \"tuned-thlx8\" (UID: \"4b78335d-1f32-4d95-adda-efac2d74e4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-thlx8" Jun 02 01:41:23.904633 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.904164 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ae3e1499-5744-4270-b94d-f292d30bd1b2-cnibin\") pod \"multus-additional-cni-plugins-tglfq\" (UID: \"ae3e1499-5744-4270-b94d-f292d30bd1b2\") " pod="openshift-multus/multus-additional-cni-plugins-tglfq" Jun 02 01:41:23.904633 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.904218 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b78335d-1f32-4d95-adda-efac2d74e4e1-tmp\") pod \"tuned-thlx8\" (UID: \"4b78335d-1f32-4d95-adda-efac2d74e4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-thlx8" Jun 02 01:41:23.904633 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.904249 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xvkvg\" (UniqueName: \"kubernetes.io/projected/4b78335d-1f32-4d95-adda-efac2d74e4e1-kube-api-access-xvkvg\") pod \"tuned-thlx8\" (UID: \"4b78335d-1f32-4d95-adda-efac2d74e4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-thlx8" Jun 02 01:41:23.904633 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.904274 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-467r4\" (UniqueName: \"kubernetes.io/projected/5c45e264-330c-4438-bdc1-817a0a8b1b77-kube-api-access-467r4\") pod \"node-resolver-nm89v\" (UID: \"5c45e264-330c-4438-bdc1-817a0a8b1b77\") " pod="openshift-dns/node-resolver-nm89v" Jun 02 01:41:23.904633 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.904297 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ba65c8da-80cd-4fda-9746-dd851a3a06be-system-cni-dir\") pod \"multus-7drhb\" (UID: \"ba65c8da-80cd-4fda-9746-dd851a3a06be\") " pod="openshift-multus/multus-7drhb" Jun 02 01:41:23.904633 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.904346 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4b78335d-1f32-4d95-adda-efac2d74e4e1-etc-sysctl-d\") pod \"tuned-thlx8\" (UID: \"4b78335d-1f32-4d95-adda-efac2d74e4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-thlx8" Jun 02 01:41:23.904633 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.904382 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ba65c8da-80cd-4fda-9746-dd851a3a06be-system-cni-dir\") pod \"multus-7drhb\" (UID: \"ba65c8da-80cd-4fda-9746-dd851a3a06be\") " pod="openshift-multus/multus-7drhb" Jun 02 01:41:23.904633 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.904386 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1fc82c63-2678-486d-9393-c3ea7f27abce-serviceca\") pod \"node-ca-6rzvj\" (UID: \"1fc82c63-2678-486d-9393-c3ea7f27abce\") " pod="openshift-image-registry/node-ca-6rzvj" Jun 02 01:41:23.904633 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.904419 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba65c8da-80cd-4fda-9746-dd851a3a06be-host-var-lib-kubelet\") pod \"multus-7drhb\" (UID: \"ba65c8da-80cd-4fda-9746-dd851a3a06be\") " pod="openshift-multus/multus-7drhb" Jun 02 01:41:23.904633 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.904449 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8f82072c-4fa6-41c2-91ce-6c7cd16a0840-host-slash\") pod \"iptables-alerter-6dnbd\" (UID: \"8f82072c-4fa6-41c2-91ce-6c7cd16a0840\") " pod="openshift-network-operator/iptables-alerter-6dnbd" Jun 02 01:41:23.904633 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.904487 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5qwx4\" (UniqueName: \"kubernetes.io/projected/1fc82c63-2678-486d-9393-c3ea7f27abce-kube-api-access-5qwx4\") pod \"node-ca-6rzvj\" (UID: \"1fc82c63-2678-486d-9393-c3ea7f27abce\") " pod="openshift-image-registry/node-ca-6rzvj" Jun 02 01:41:23.905238 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.904489 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba65c8da-80cd-4fda-9746-dd851a3a06be-host-var-lib-kubelet\") pod \"multus-7drhb\" (UID: \"ba65c8da-80cd-4fda-9746-dd851a3a06be\") " pod="openshift-multus/multus-7drhb" Jun 02 01:41:23.905238 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.904522 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ab3aef63-c4c0-4b01-b26b-53ca0e39e7bb-konnectivity-ca\") pod \"konnectivity-agent-bmd65\" (UID: \"ab3aef63-c4c0-4b01-b26b-53ca0e39e7bb\") " pod="kube-system/konnectivity-agent-bmd65" Jun 02 01:41:23.905238 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.904534 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8f82072c-4fa6-41c2-91ce-6c7cd16a0840-host-slash\") pod \"iptables-alerter-6dnbd\" (UID: \"8f82072c-4fa6-41c2-91ce-6c7cd16a0840\") " pod="openshift-network-operator/iptables-alerter-6dnbd" Jun 02 01:41:23.905238 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.904553 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aa0d7d15-d800-4698-a39f-763402e48110-host-slash\") pod \"ovnkube-node-fvdb7\" (UID: \"aa0d7d15-d800-4698-a39f-763402e48110\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" Jun 02 01:41:23.905238 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.904537 2571 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jun 02 01:41:23.905238 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.904572 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3542744-21d2-4162-a2f9-4a474d18dc19-metrics-certs\") pod \"network-metrics-daemon-smqwh\" (UID: \"f3542744-21d2-4162-a2f9-4a474d18dc19\") " pod="openshift-multus/network-metrics-daemon-smqwh" Jun 02 01:41:23.905238 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.904588 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5d9fc\" (UniqueName: \"kubernetes.io/projected/f3542744-21d2-4162-a2f9-4a474d18dc19-kube-api-access-5d9fc\") pod \"network-metrics-daemon-smqwh\" (UID: \"f3542744-21d2-4162-a2f9-4a474d18dc19\") " pod="openshift-multus/network-metrics-daemon-smqwh" Jun 02 01:41:23.905238 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.904611 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-m4bbk\" (UniqueName: \"kubernetes.io/projected/8f82072c-4fa6-41c2-91ce-6c7cd16a0840-kube-api-access-m4bbk\") pod \"iptables-alerter-6dnbd\" (UID: \"8f82072c-4fa6-41c2-91ce-6c7cd16a0840\") " pod="openshift-network-operator/iptables-alerter-6dnbd" Jun 02 01:41:23.905238 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.904628 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b78335d-1f32-4d95-adda-efac2d74e4e1-sys\") pod \"tuned-thlx8\" (UID: \"4b78335d-1f32-4d95-adda-efac2d74e4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-thlx8" Jun 02 01:41:23.905238 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.904630 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aa0d7d15-d800-4698-a39f-763402e48110-host-slash\") pod \"ovnkube-node-fvdb7\" (UID: \"aa0d7d15-d800-4698-a39f-763402e48110\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" Jun 02 01:41:23.905238 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.904643 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ae3e1499-5744-4270-b94d-f292d30bd1b2-os-release\") pod \"multus-additional-cni-plugins-tglfq\" (UID: \"ae3e1499-5744-4270-b94d-f292d30bd1b2\") " pod="openshift-multus/multus-additional-cni-plugins-tglfq" Jun 02 01:41:23.905238 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.904663 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ae3e1499-5744-4270-b94d-f292d30bd1b2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tglfq\" (UID: \"ae3e1499-5744-4270-b94d-f292d30bd1b2\") " pod="openshift-multus/multus-additional-cni-plugins-tglfq" Jun 02 01:41:23.905238 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:23.904667 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jun 02 01:41:23.905238 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.904694 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5cffe19e-fcbd-46bb-9db3-2617a77f95f6-registration-dir\") pod \"aws-ebs-csi-driver-node-7zf8v\" (UID: \"5cffe19e-fcbd-46bb-9db3-2617a77f95f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zf8v" Jun 02 01:41:23.905238 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.904719 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ba65c8da-80cd-4fda-9746-dd851a3a06be-cni-binary-copy\") pod \"multus-7drhb\" (UID: \"ba65c8da-80cd-4fda-9746-dd851a3a06be\") " pod="openshift-multus/multus-7drhb" Jun 02 01:41:23.905238 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.904734 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ae3e1499-5744-4270-b94d-f292d30bd1b2-os-release\") pod \"multus-additional-cni-plugins-tglfq\" (UID: \"ae3e1499-5744-4270-b94d-f292d30bd1b2\") " pod="openshift-multus/multus-additional-cni-plugins-tglfq" Jun 02 01:41:23.905238 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:23.904746 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3542744-21d2-4162-a2f9-4a474d18dc19-metrics-certs podName:f3542744-21d2-4162-a2f9-4a474d18dc19 nodeName:}" failed. No retries permitted until 2026-06-02 01:41:24.404716871 +0000 UTC m=+2.050523507 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f3542744-21d2-4162-a2f9-4a474d18dc19-metrics-certs") pod "network-metrics-daemon-smqwh" (UID: "f3542744-21d2-4162-a2f9-4a474d18dc19") : object "openshift-multus"/"metrics-daemon-secret" not registered Jun 02 01:41:23.905869 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.904784 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aa0d7d15-d800-4698-a39f-763402e48110-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fvdb7\" (UID: \"aa0d7d15-d800-4698-a39f-763402e48110\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" Jun 02 01:41:23.905869 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.904815 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/aa0d7d15-d800-4698-a39f-763402e48110-host-kubelet\") pod \"ovnkube-node-fvdb7\" (UID: \"aa0d7d15-d800-4698-a39f-763402e48110\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" Jun 02 01:41:23.905869 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.904858 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/aa0d7d15-d800-4698-a39f-763402e48110-host-kubelet\") pod \"ovnkube-node-fvdb7\" (UID: \"aa0d7d15-d800-4698-a39f-763402e48110\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" Jun 02 01:41:23.905869 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.904862 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b78335d-1f32-4d95-adda-efac2d74e4e1-sys\") pod \"tuned-thlx8\" (UID: \"4b78335d-1f32-4d95-adda-efac2d74e4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-thlx8" Jun 02 01:41:23.905869 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.904872 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5cffe19e-fcbd-46bb-9db3-2617a77f95f6-registration-dir\") pod \"aws-ebs-csi-driver-node-7zf8v\" (UID: \"5cffe19e-fcbd-46bb-9db3-2617a77f95f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zf8v" Jun 02 01:41:23.905869 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.904897 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aa0d7d15-d800-4698-a39f-763402e48110-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fvdb7\" (UID: \"aa0d7d15-d800-4698-a39f-763402e48110\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" Jun 02 01:41:23.905869 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.904903 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aa0d7d15-d800-4698-a39f-763402e48110-host-run-netns\") pod \"ovnkube-node-fvdb7\" (UID: \"aa0d7d15-d800-4698-a39f-763402e48110\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" Jun 02 01:41:23.905869 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.904895 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ae3e1499-5744-4270-b94d-f292d30bd1b2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tglfq\" (UID: \"ae3e1499-5744-4270-b94d-f292d30bd1b2\") " pod="openshift-multus/multus-additional-cni-plugins-tglfq" Jun 02 01:41:23.905869 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.904943 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aa0d7d15-d800-4698-a39f-763402e48110-etc-openvswitch\") pod \"ovnkube-node-fvdb7\" (UID: \"aa0d7d15-d800-4698-a39f-763402e48110\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" Jun 02 01:41:23.905869 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.904969 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aa0d7d15-d800-4698-a39f-763402e48110-run-openvswitch\") pod \"ovnkube-node-fvdb7\" (UID: \"aa0d7d15-d800-4698-a39f-763402e48110\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" Jun 02 01:41:23.905869 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.904978 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aa0d7d15-d800-4698-a39f-763402e48110-host-run-netns\") pod \"ovnkube-node-fvdb7\" (UID: \"aa0d7d15-d800-4698-a39f-763402e48110\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" Jun 02 01:41:23.905869 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.905003 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aa0d7d15-d800-4698-a39f-763402e48110-host-run-ovn-kubernetes\") pod \"ovnkube-node-fvdb7\" (UID: \"aa0d7d15-d800-4698-a39f-763402e48110\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" Jun 02 01:41:23.905869 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.905013 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aa0d7d15-d800-4698-a39f-763402e48110-etc-openvswitch\") pod \"ovnkube-node-fvdb7\" (UID: \"aa0d7d15-d800-4698-a39f-763402e48110\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" Jun 02 01:41:23.905869 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.905025 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aa0d7d15-d800-4698-a39f-763402e48110-run-openvswitch\") pod \"ovnkube-node-fvdb7\" (UID: \"aa0d7d15-d800-4698-a39f-763402e48110\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" Jun 02 01:41:23.905869 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.905030 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5cffe19e-fcbd-46bb-9db3-2617a77f95f6-socket-dir\") pod \"aws-ebs-csi-driver-node-7zf8v\" (UID: \"5cffe19e-fcbd-46bb-9db3-2617a77f95f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zf8v" Jun 02 01:41:23.905869 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.905055 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ba65c8da-80cd-4fda-9746-dd851a3a06be-multus-conf-dir\") pod \"multus-7drhb\" (UID: \"ba65c8da-80cd-4fda-9746-dd851a3a06be\") " pod="openshift-multus/multus-7drhb" Jun 02 01:41:23.905869 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.905070 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aa0d7d15-d800-4698-a39f-763402e48110-host-run-ovn-kubernetes\") pod \"ovnkube-node-fvdb7\" (UID: \"aa0d7d15-d800-4698-a39f-763402e48110\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" Jun 02 01:41:23.906583 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.905079 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ae3e1499-5744-4270-b94d-f292d30bd1b2-cni-binary-copy\") pod \"multus-additional-cni-plugins-tglfq\" (UID: \"ae3e1499-5744-4270-b94d-f292d30bd1b2\") " pod="openshift-multus/multus-additional-cni-plugins-tglfq" Jun 02 01:41:23.906583 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.905095 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/aa0d7d15-d800-4698-a39f-763402e48110-host-cni-netd\") pod \"ovnkube-node-fvdb7\" (UID: \"aa0d7d15-d800-4698-a39f-763402e48110\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" Jun 02 01:41:23.906583 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.905119 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ba65c8da-80cd-4fda-9746-dd851a3a06be-multus-daemon-config\") pod \"multus-7drhb\" (UID: \"ba65c8da-80cd-4fda-9746-dd851a3a06be\") " pod="openshift-multus/multus-7drhb" Jun 02 01:41:23.906583 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.905118 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ba65c8da-80cd-4fda-9746-dd851a3a06be-multus-conf-dir\") pod \"multus-7drhb\" (UID: \"ba65c8da-80cd-4fda-9746-dd851a3a06be\") " pod="openshift-multus/multus-7drhb" Jun 02 01:41:23.906583 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.905173 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b78335d-1f32-4d95-adda-efac2d74e4e1-var-lib-kubelet\") pod \"tuned-thlx8\" (UID: \"4b78335d-1f32-4d95-adda-efac2d74e4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-thlx8" Jun 02 01:41:23.906583 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.905176 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5cffe19e-fcbd-46bb-9db3-2617a77f95f6-socket-dir\") pod \"aws-ebs-csi-driver-node-7zf8v\" (UID: \"5cffe19e-fcbd-46bb-9db3-2617a77f95f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zf8v" Jun 02 01:41:23.906583 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.905206 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ae3e1499-5744-4270-b94d-f292d30bd1b2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tglfq\" (UID: \"ae3e1499-5744-4270-b94d-f292d30bd1b2\") " pod="openshift-multus/multus-additional-cni-plugins-tglfq" Jun 02 01:41:23.906583 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.905235 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ba65c8da-80cd-4fda-9746-dd851a3a06be-multus-cni-dir\") pod \"multus-7drhb\" (UID: \"ba65c8da-80cd-4fda-9746-dd851a3a06be\") " pod="openshift-multus/multus-7drhb" Jun 02 01:41:23.906583 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.905243 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/aa0d7d15-d800-4698-a39f-763402e48110-host-cni-netd\") pod \"ovnkube-node-fvdb7\" (UID: \"aa0d7d15-d800-4698-a39f-763402e48110\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" Jun 02 01:41:23.906583 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.905258 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ba65c8da-80cd-4fda-9746-dd851a3a06be-multus-socket-dir-parent\") pod \"multus-7drhb\" (UID: \"ba65c8da-80cd-4fda-9746-dd851a3a06be\") " pod="openshift-multus/multus-7drhb" Jun 02 01:41:23.906583 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.905274 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ba65c8da-80cd-4fda-9746-dd851a3a06be-host-run-k8s-cni-cncf-io\") pod \"multus-7drhb\" (UID: \"ba65c8da-80cd-4fda-9746-dd851a3a06be\") " pod="openshift-multus/multus-7drhb" Jun 02 01:41:23.906583 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.905289 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ba65c8da-80cd-4fda-9746-dd851a3a06be-host-var-lib-cni-bin\") pod \"multus-7drhb\" (UID: \"ba65c8da-80cd-4fda-9746-dd851a3a06be\") " pod="openshift-multus/multus-7drhb" Jun 02 01:41:23.906583 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.905306 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4b78335d-1f32-4d95-adda-efac2d74e4e1-etc-systemd\") pod \"tuned-thlx8\" (UID: \"4b78335d-1f32-4d95-adda-efac2d74e4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-thlx8" Jun 02 01:41:23.906583 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.905321 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4b78335d-1f32-4d95-adda-efac2d74e4e1-etc-tuned\") pod \"tuned-thlx8\" (UID: \"4b78335d-1f32-4d95-adda-efac2d74e4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-thlx8" Jun 02 01:41:23.906583 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.905337 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c45e264-330c-4438-bdc1-817a0a8b1b77-tmp-dir\") pod \"node-resolver-nm89v\" (UID: \"5c45e264-330c-4438-bdc1-817a0a8b1b77\") " pod="openshift-dns/node-resolver-nm89v" Jun 02 01:41:23.906583 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.905351 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5cffe19e-fcbd-46bb-9db3-2617a77f95f6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7zf8v\" (UID: \"5cffe19e-fcbd-46bb-9db3-2617a77f95f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zf8v" Jun 02 01:41:23.906583 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.905366 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5cffe19e-fcbd-46bb-9db3-2617a77f95f6-sys-fs\") pod \"aws-ebs-csi-driver-node-7zf8v\" (UID: \"5cffe19e-fcbd-46bb-9db3-2617a77f95f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zf8v" Jun 02 01:41:23.907151 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.905381 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4b78335d-1f32-4d95-adda-efac2d74e4e1-etc-sysctl-conf\") pod \"tuned-thlx8\" (UID: \"4b78335d-1f32-4d95-adda-efac2d74e4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-thlx8" Jun 02 01:41:23.907151 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.905398 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wccb2\" (UniqueName: \"kubernetes.io/projected/ae3e1499-5744-4270-b94d-f292d30bd1b2-kube-api-access-wccb2\") pod \"multus-additional-cni-plugins-tglfq\" (UID: \"ae3e1499-5744-4270-b94d-f292d30bd1b2\") " pod="openshift-multus/multus-additional-cni-plugins-tglfq" Jun 02 01:41:23.907151 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.905415 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ba65c8da-80cd-4fda-9746-dd851a3a06be-host-run-multus-certs\") pod \"multus-7drhb\" (UID: \"ba65c8da-80cd-4fda-9746-dd851a3a06be\") " pod="openshift-multus/multus-7drhb" Jun 02 01:41:23.907151 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.905430 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4b78335d-1f32-4d95-adda-efac2d74e4e1-host\") pod \"tuned-thlx8\" (UID: \"4b78335d-1f32-4d95-adda-efac2d74e4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-thlx8" Jun 02 01:41:23.907151 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.905446 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/aa0d7d15-d800-4698-a39f-763402e48110-node-log\") pod \"ovnkube-node-fvdb7\" (UID: \"aa0d7d15-d800-4698-a39f-763402e48110\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" Jun 02 01:41:23.907151 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.905461 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/aa0d7d15-d800-4698-a39f-763402e48110-log-socket\") pod \"ovnkube-node-fvdb7\" (UID: \"aa0d7d15-d800-4698-a39f-763402e48110\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" Jun 02 01:41:23.907151 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.905477 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/aa0d7d15-d800-4698-a39f-763402e48110-ovnkube-config\") pod \"ovnkube-node-fvdb7\" (UID: \"aa0d7d15-d800-4698-a39f-763402e48110\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" Jun 02 01:41:23.907151 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.905493 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/aa0d7d15-d800-4698-a39f-763402e48110-ovnkube-script-lib\") pod \"ovnkube-node-fvdb7\" (UID: \"aa0d7d15-d800-4698-a39f-763402e48110\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" Jun 02 01:41:23.907151 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.905508 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5c45e264-330c-4438-bdc1-817a0a8b1b77-hosts-file\") pod \"node-resolver-nm89v\" (UID: \"5c45e264-330c-4438-bdc1-817a0a8b1b77\") " pod="openshift-dns/node-resolver-nm89v" Jun 02 01:41:23.907151 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.905525 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ba65c8da-80cd-4fda-9746-dd851a3a06be-host-var-lib-cni-multus\") pod \"multus-7drhb\" (UID: \"ba65c8da-80cd-4fda-9746-dd851a3a06be\") " pod="openshift-multus/multus-7drhb" Jun 02 01:41:23.907151 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.905540 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/aa0d7d15-d800-4698-a39f-763402e48110-systemd-units\") pod \"ovnkube-node-fvdb7\" (UID: \"aa0d7d15-d800-4698-a39f-763402e48110\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" Jun 02 01:41:23.907151 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.905555 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/aa0d7d15-d800-4698-a39f-763402e48110-run-ovn\") pod \"ovnkube-node-fvdb7\" (UID: \"aa0d7d15-d800-4698-a39f-763402e48110\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" Jun 02 01:41:23.907151 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.905571 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4b78335d-1f32-4d95-adda-efac2d74e4e1-etc-sysconfig\") pod \"tuned-thlx8\" (UID: \"4b78335d-1f32-4d95-adda-efac2d74e4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-thlx8" Jun 02 01:41:23.907151 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.905580 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ba65c8da-80cd-4fda-9746-dd851a3a06be-multus-daemon-config\") pod \"multus-7drhb\" (UID: \"ba65c8da-80cd-4fda-9746-dd851a3a06be\") " pod="openshift-multus/multus-7drhb" Jun 02 01:41:23.907151 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.905614 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ae3e1499-5744-4270-b94d-f292d30bd1b2-cni-binary-copy\") pod \"multus-additional-cni-plugins-tglfq\" (UID: \"ae3e1499-5744-4270-b94d-f292d30bd1b2\") " pod="openshift-multus/multus-additional-cni-plugins-tglfq" Jun 02 01:41:23.907151 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.905648 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ae3e1499-5744-4270-b94d-f292d30bd1b2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tglfq\" (UID: \"ae3e1499-5744-4270-b94d-f292d30bd1b2\") " pod="openshift-multus/multus-additional-cni-plugins-tglfq" Jun 02 01:41:23.907151 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.905690 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4b78335d-1f32-4d95-adda-efac2d74e4e1-etc-sysctl-conf\") pod \"tuned-thlx8\" (UID: \"4b78335d-1f32-4d95-adda-efac2d74e4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-thlx8" Jun 02 01:41:23.907654 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.905740 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/aa0d7d15-d800-4698-a39f-763402e48110-log-socket\") pod \"ovnkube-node-fvdb7\" (UID: \"aa0d7d15-d800-4698-a39f-763402e48110\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" Jun 02 01:41:23.907654 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.905694 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4b78335d-1f32-4d95-adda-efac2d74e4e1-lib-modules\") pod \"tuned-thlx8\" (UID: \"4b78335d-1f32-4d95-adda-efac2d74e4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-thlx8" Jun 02 01:41:23.907654 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.905694 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4b78335d-1f32-4d95-adda-efac2d74e4e1-host\") pod \"tuned-thlx8\" (UID: \"4b78335d-1f32-4d95-adda-efac2d74e4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-thlx8" Jun 02 01:41:23.907654 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.905586 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4b78335d-1f32-4d95-adda-efac2d74e4e1-lib-modules\") pod \"tuned-thlx8\" (UID: \"4b78335d-1f32-4d95-adda-efac2d74e4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-thlx8" Jun 02 01:41:23.907654 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.905769 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5cffe19e-fcbd-46bb-9db3-2617a77f95f6-sys-fs\") pod \"aws-ebs-csi-driver-node-7zf8v\" (UID: \"5cffe19e-fcbd-46bb-9db3-2617a77f95f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zf8v" Jun 02 01:41:23.907654 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.905792 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/aa0d7d15-d800-4698-a39f-763402e48110-env-overrides\") pod \"ovnkube-node-fvdb7\" (UID: \"aa0d7d15-d800-4698-a39f-763402e48110\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" Jun 02 01:41:23.907654 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.905693 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b78335d-1f32-4d95-adda-efac2d74e4e1-var-lib-kubelet\") pod \"tuned-thlx8\" (UID: \"4b78335d-1f32-4d95-adda-efac2d74e4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-thlx8" Jun 02 01:41:23.907654 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.905813 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ba65c8da-80cd-4fda-9746-dd851a3a06be-cni-binary-copy\") pod \"multus-7drhb\" (UID: \"ba65c8da-80cd-4fda-9746-dd851a3a06be\") " pod="openshift-multus/multus-7drhb" Jun 02 01:41:23.907654 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.905841 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ba65c8da-80cd-4fda-9746-dd851a3a06be-os-release\") pod \"multus-7drhb\" (UID: \"ba65c8da-80cd-4fda-9746-dd851a3a06be\") " pod="openshift-multus/multus-7drhb" Jun 02 01:41:23.907654 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.905854 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5cffe19e-fcbd-46bb-9db3-2617a77f95f6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7zf8v\" (UID: \"5cffe19e-fcbd-46bb-9db3-2617a77f95f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zf8v" Jun 02 01:41:23.907654 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.905892 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/aa0d7d15-d800-4698-a39f-763402e48110-node-log\") pod \"ovnkube-node-fvdb7\" (UID: \"aa0d7d15-d800-4698-a39f-763402e48110\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" Jun 02 01:41:23.907654 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.905892 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ba65c8da-80cd-4fda-9746-dd851a3a06be-host-run-multus-certs\") pod \"multus-7drhb\" (UID: \"ba65c8da-80cd-4fda-9746-dd851a3a06be\") " pod="openshift-multus/multus-7drhb" Jun 02 01:41:23.907654 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.905924 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ba65c8da-80cd-4fda-9746-dd851a3a06be-os-release\") pod \"multus-7drhb\" (UID: \"ba65c8da-80cd-4fda-9746-dd851a3a06be\") " pod="openshift-multus/multus-7drhb" Jun 02 01:41:23.907654 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.905926 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ba65c8da-80cd-4fda-9746-dd851a3a06be-host-run-k8s-cni-cncf-io\") pod \"multus-7drhb\" (UID: \"ba65c8da-80cd-4fda-9746-dd851a3a06be\") " pod="openshift-multus/multus-7drhb" Jun 02 01:41:23.907654 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.905975 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5kbsh\" (UniqueName: \"kubernetes.io/projected/ba65c8da-80cd-4fda-9746-dd851a3a06be-kube-api-access-5kbsh\") pod \"multus-7drhb\" (UID: \"ba65c8da-80cd-4fda-9746-dd851a3a06be\") " pod="openshift-multus/multus-7drhb" Jun 02 01:41:23.907654 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.906009 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4b78335d-1f32-4d95-adda-efac2d74e4e1-etc-sysconfig\") pod \"tuned-thlx8\" (UID: \"4b78335d-1f32-4d95-adda-efac2d74e4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-thlx8" Jun 02 01:41:23.907654 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.906048 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ba65c8da-80cd-4fda-9746-dd851a3a06be-multus-cni-dir\") pod \"multus-7drhb\" (UID: \"ba65c8da-80cd-4fda-9746-dd851a3a06be\") " pod="openshift-multus/multus-7drhb" Jun 02 01:41:23.907654 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.906082 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ba65c8da-80cd-4fda-9746-dd851a3a06be-host-var-lib-cni-bin\") pod \"multus-7drhb\" (UID: \"ba65c8da-80cd-4fda-9746-dd851a3a06be\") " pod="openshift-multus/multus-7drhb" Jun 02 01:41:23.908187 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.906094 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4b78335d-1f32-4d95-adda-efac2d74e4e1-etc-systemd\") pod \"tuned-thlx8\" (UID: \"4b78335d-1f32-4d95-adda-efac2d74e4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-thlx8" Jun 02 01:41:23.908187 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.906114 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8f82072c-4fa6-41c2-91ce-6c7cd16a0840-iptables-alerter-script\") pod \"iptables-alerter-6dnbd\" (UID: \"8f82072c-4fa6-41c2-91ce-6c7cd16a0840\") " pod="openshift-network-operator/iptables-alerter-6dnbd" Jun 02 01:41:23.908187 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.906144 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/aa0d7d15-d800-4698-a39f-763402e48110-run-ovn\") pod \"ovnkube-node-fvdb7\" (UID: \"aa0d7d15-d800-4698-a39f-763402e48110\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" Jun 02 01:41:23.908187 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.906164 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4b78335d-1f32-4d95-adda-efac2d74e4e1-run\") pod \"tuned-thlx8\" (UID: \"4b78335d-1f32-4d95-adda-efac2d74e4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-thlx8" Jun 02 01:41:23.908187 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.906181 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/aa0d7d15-d800-4698-a39f-763402e48110-ovnkube-script-lib\") pod \"ovnkube-node-fvdb7\" (UID: \"aa0d7d15-d800-4698-a39f-763402e48110\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" Jun 02 01:41:23.908187 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.906187 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ba65c8da-80cd-4fda-9746-dd851a3a06be-host-var-lib-cni-multus\") pod \"multus-7drhb\" (UID: \"ba65c8da-80cd-4fda-9746-dd851a3a06be\") " pod="openshift-multus/multus-7drhb" Jun 02 01:41:23.908187 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.906194 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ae3e1499-5744-4270-b94d-f292d30bd1b2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tglfq\" (UID: \"ae3e1499-5744-4270-b94d-f292d30bd1b2\") " pod="openshift-multus/multus-additional-cni-plugins-tglfq" Jun 02 01:41:23.908187 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.906223 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/aa0d7d15-d800-4698-a39f-763402e48110-run-systemd\") pod \"ovnkube-node-fvdb7\" (UID: \"aa0d7d15-d800-4698-a39f-763402e48110\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" Jun 02 01:41:23.908187 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.906228 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5c45e264-330c-4438-bdc1-817a0a8b1b77-hosts-file\") pod \"node-resolver-nm89v\" (UID: \"5c45e264-330c-4438-bdc1-817a0a8b1b77\") " pod="openshift-dns/node-resolver-nm89v" Jun 02 01:41:23.908187 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.906264 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/aa0d7d15-d800-4698-a39f-763402e48110-systemd-units\") pod \"ovnkube-node-fvdb7\" (UID: \"aa0d7d15-d800-4698-a39f-763402e48110\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" Jun 02 01:41:23.908187 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.906267 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/aa0d7d15-d800-4698-a39f-763402e48110-run-systemd\") pod \"ovnkube-node-fvdb7\" (UID: \"aa0d7d15-d800-4698-a39f-763402e48110\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" Jun 02 01:41:23.908187 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.906370 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/aa0d7d15-d800-4698-a39f-763402e48110-ovnkube-config\") pod \"ovnkube-node-fvdb7\" (UID: \"aa0d7d15-d800-4698-a39f-763402e48110\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" Jun 02 01:41:23.908187 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.906386 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/aa0d7d15-d800-4698-a39f-763402e48110-env-overrides\") pod \"ovnkube-node-fvdb7\" (UID: \"aa0d7d15-d800-4698-a39f-763402e48110\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" Jun 02 01:41:23.908187 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.906435 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4b78335d-1f32-4d95-adda-efac2d74e4e1-run\") pod \"tuned-thlx8\" (UID: \"4b78335d-1f32-4d95-adda-efac2d74e4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-thlx8" Jun 02 01:41:23.908187 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.906437 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aa0d7d15-d800-4698-a39f-763402e48110-var-lib-openvswitch\") pod \"ovnkube-node-fvdb7\" (UID: \"aa0d7d15-d800-4698-a39f-763402e48110\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" Jun 02 01:41:23.908187 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.906477 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aa0d7d15-d800-4698-a39f-763402e48110-var-lib-openvswitch\") pod \"ovnkube-node-fvdb7\" (UID: \"aa0d7d15-d800-4698-a39f-763402e48110\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" Jun 02 01:41:23.908187 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.906492 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ba65c8da-80cd-4fda-9746-dd851a3a06be-multus-socket-dir-parent\") pod \"multus-7drhb\" (UID: \"ba65c8da-80cd-4fda-9746-dd851a3a06be\") " pod="openshift-multus/multus-7drhb" Jun 02 01:41:23.908187 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.906589 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5cffe19e-fcbd-46bb-9db3-2617a77f95f6-device-dir\") pod \"aws-ebs-csi-driver-node-7zf8v\" (UID: \"5cffe19e-fcbd-46bb-9db3-2617a77f95f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zf8v" Jun 02 01:41:23.908752 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.906693 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5cffe19e-fcbd-46bb-9db3-2617a77f95f6-device-dir\") pod \"aws-ebs-csi-driver-node-7zf8v\" (UID: \"5cffe19e-fcbd-46bb-9db3-2617a77f95f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zf8v" Jun 02 01:41:23.908752 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.906734 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8f82072c-4fa6-41c2-91ce-6c7cd16a0840-iptables-alerter-script\") pod \"iptables-alerter-6dnbd\" (UID: \"8f82072c-4fa6-41c2-91ce-6c7cd16a0840\") " pod="openshift-network-operator/iptables-alerter-6dnbd" Jun 02 01:41:23.908752 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.906756 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ae3e1499-5744-4270-b94d-f292d30bd1b2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tglfq\" (UID: \"ae3e1499-5744-4270-b94d-f292d30bd1b2\") " pod="openshift-multus/multus-additional-cni-plugins-tglfq" Jun 02 01:41:23.908752 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.906926 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c45e264-330c-4438-bdc1-817a0a8b1b77-tmp-dir\") pod \"node-resolver-nm89v\" (UID: \"5c45e264-330c-4438-bdc1-817a0a8b1b77\") " pod="openshift-dns/node-resolver-nm89v" Jun 02 01:41:23.908752 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.908201 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b78335d-1f32-4d95-adda-efac2d74e4e1-tmp\") pod \"tuned-thlx8\" (UID: \"4b78335d-1f32-4d95-adda-efac2d74e4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-thlx8" Jun 02 01:41:23.908752 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.908204 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/aa0d7d15-d800-4698-a39f-763402e48110-ovn-node-metrics-cert\") pod \"ovnkube-node-fvdb7\" (UID: \"aa0d7d15-d800-4698-a39f-763402e48110\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" Jun 02 01:41:23.908752 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.908509 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4b78335d-1f32-4d95-adda-efac2d74e4e1-etc-tuned\") pod \"tuned-thlx8\" (UID: \"4b78335d-1f32-4d95-adda-efac2d74e4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-thlx8" Jun 02 01:41:23.911839 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:23.911824 2571 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jun 02 01:41:23.911903 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:23.911844 2571 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jun 02 01:41:23.911903 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:23.911853 2571 projected.go:194] Error preparing data for projected volume kube-api-access-6f7hb for pod openshift-network-diagnostics/network-check-target-gt4w8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 02 01:41:23.911903 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:23.911898 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7db70fbd-3197-47ed-a00b-d6b4fbb4c8a8-kube-api-access-6f7hb podName:7db70fbd-3197-47ed-a00b-d6b4fbb4c8a8 nodeName:}" failed. No retries permitted until 2026-06-02 01:41:24.411883762 +0000 UTC m=+2.057690385 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-6f7hb" (UniqueName: "kubernetes.io/projected/7db70fbd-3197-47ed-a00b-d6b4fbb4c8a8-kube-api-access-6f7hb") pod "network-check-target-gt4w8" (UID: "7db70fbd-3197-47ed-a00b-d6b4fbb4c8a8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 02 01:41:23.914185 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.914163 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4bbk\" (UniqueName: \"kubernetes.io/projected/8f82072c-4fa6-41c2-91ce-6c7cd16a0840-kube-api-access-m4bbk\") pod \"iptables-alerter-6dnbd\" (UID: \"8f82072c-4fa6-41c2-91ce-6c7cd16a0840\") " pod="openshift-network-operator/iptables-alerter-6dnbd" Jun 02 01:41:23.914865 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.914825 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtw5j\" (UniqueName: \"kubernetes.io/projected/aa0d7d15-d800-4698-a39f-763402e48110-kube-api-access-xtw5j\") pod \"ovnkube-node-fvdb7\" (UID: \"aa0d7d15-d800-4698-a39f-763402e48110\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" Jun 02 01:41:23.915495 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.915056 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qwx4\" (UniqueName: \"kubernetes.io/projected/1fc82c63-2678-486d-9393-c3ea7f27abce-kube-api-access-5qwx4\") pod \"node-ca-6rzvj\" (UID: \"1fc82c63-2678-486d-9393-c3ea7f27abce\") " pod="openshift-image-registry/node-ca-6rzvj" Jun 02 01:41:23.915495 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.915477 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvkvg\" (UniqueName: \"kubernetes.io/projected/4b78335d-1f32-4d95-adda-efac2d74e4e1-kube-api-access-xvkvg\") pod \"tuned-thlx8\" (UID: \"4b78335d-1f32-4d95-adda-efac2d74e4e1\") " pod="openshift-cluster-node-tuning-operator/tuned-thlx8" Jun 02 01:41:23.915631 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.915475 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh8f2\" (UniqueName: \"kubernetes.io/projected/5cffe19e-fcbd-46bb-9db3-2617a77f95f6-kube-api-access-qh8f2\") pod \"aws-ebs-csi-driver-node-7zf8v\" (UID: \"5cffe19e-fcbd-46bb-9db3-2617a77f95f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zf8v" Jun 02 01:41:23.915755 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.915720 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d9fc\" (UniqueName: \"kubernetes.io/projected/f3542744-21d2-4162-a2f9-4a474d18dc19-kube-api-access-5d9fc\") pod \"network-metrics-daemon-smqwh\" (UID: \"f3542744-21d2-4162-a2f9-4a474d18dc19\") " pod="openshift-multus/network-metrics-daemon-smqwh" Jun 02 01:41:23.915938 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.915921 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kbsh\" (UniqueName: \"kubernetes.io/projected/ba65c8da-80cd-4fda-9746-dd851a3a06be-kube-api-access-5kbsh\") pod \"multus-7drhb\" (UID: \"ba65c8da-80cd-4fda-9746-dd851a3a06be\") " pod="openshift-multus/multus-7drhb" Jun 02 01:41:23.916162 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.916122 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wccb2\" (UniqueName: \"kubernetes.io/projected/ae3e1499-5744-4270-b94d-f292d30bd1b2-kube-api-access-wccb2\") pod \"multus-additional-cni-plugins-tglfq\" (UID: \"ae3e1499-5744-4270-b94d-f292d30bd1b2\") " pod="openshift-multus/multus-additional-cni-plugins-tglfq" Jun 02 01:41:23.916492 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.916478 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-467r4\" (UniqueName: \"kubernetes.io/projected/5c45e264-330c-4438-bdc1-817a0a8b1b77-kube-api-access-467r4\") pod \"node-resolver-nm89v\" (UID: \"5c45e264-330c-4438-bdc1-817a0a8b1b77\") " pod="openshift-dns/node-resolver-nm89v" Jun 02 01:41:23.926189 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.926103 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-117.ec2.internal" event={"ID":"dddc001fa74725ad60cc8708aa217f2d","Type":"ContainerStarted","Data":"9e4e2c22b00e1bb1a54bc1f897b720b20bb6e6a92284d6d75d431329472a4bfb"} Jun 02 01:41:23.927032 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:23.927011 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-117.ec2.internal" event={"ID":"ee07ddb5f970db1345c2560076c9a2b8","Type":"ContainerStarted","Data":"864eb063f0e598776ff1d4f09119033898db53ef5bbbd554343f6fd06f6d6c27"} Jun 02 01:41:24.007155 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:24.007062 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ab3aef63-c4c0-4b01-b26b-53ca0e39e7bb-agent-certs\") pod \"konnectivity-agent-bmd65\" (UID: \"ab3aef63-c4c0-4b01-b26b-53ca0e39e7bb\") " pod="kube-system/konnectivity-agent-bmd65" Jun 02 01:41:24.007155 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:24.007113 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ab3aef63-c4c0-4b01-b26b-53ca0e39e7bb-konnectivity-ca\") pod \"konnectivity-agent-bmd65\" (UID: \"ab3aef63-c4c0-4b01-b26b-53ca0e39e7bb\") " pod="kube-system/konnectivity-agent-bmd65" Jun 02 01:41:24.007593 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:24.007574 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ab3aef63-c4c0-4b01-b26b-53ca0e39e7bb-konnectivity-ca\") pod \"konnectivity-agent-bmd65\" (UID: \"ab3aef63-c4c0-4b01-b26b-53ca0e39e7bb\") " pod="kube-system/konnectivity-agent-bmd65" Jun 02 01:41:24.009299 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:24.009282 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ab3aef63-c4c0-4b01-b26b-53ca0e39e7bb-agent-certs\") pod \"konnectivity-agent-bmd65\" (UID: \"ab3aef63-c4c0-4b01-b26b-53ca0e39e7bb\") " pod="kube-system/konnectivity-agent-bmd65" Jun 02 01:41:24.111413 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:24.111386 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-thlx8" Jun 02 01:41:24.117106 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:24.117081 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b78335d_1f32_4d95_adda_efac2d74e4e1.slice/crio-7ab1af8481a0b35f003c4cde449ba3ee4e3e077aeaddf71707d449d2b6d6688c WatchSource:0}: Error finding container 7ab1af8481a0b35f003c4cde449ba3ee4e3e077aeaddf71707d449d2b6d6688c: Status 404 returned error can't find the container with id 7ab1af8481a0b35f003c4cde449ba3ee4e3e077aeaddf71707d449d2b6d6688c Jun 02 01:41:24.125538 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:24.125518 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zf8v" Jun 02 01:41:24.132547 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:24.132518 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cffe19e_fcbd_46bb_9db3_2617a77f95f6.slice/crio-1169e7aef6aab904d368b443a10e6b557130b587366574843139b84570c5bc4e WatchSource:0}: Error finding container 1169e7aef6aab904d368b443a10e6b557130b587366574843139b84570c5bc4e: Status 404 returned error can't find the container with id 1169e7aef6aab904d368b443a10e6b557130b587366574843139b84570c5bc4e Jun 02 01:41:24.145689 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:24.145666 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6rzvj" Jun 02 01:41:24.151542 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:24.151521 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fc82c63_2678_486d_9393_c3ea7f27abce.slice/crio-f6e6b962d50a718098bf2af96f094ad7b26723de576d88101109078480742cdc WatchSource:0}: Error finding container f6e6b962d50a718098bf2af96f094ad7b26723de576d88101109078480742cdc: Status 404 returned error can't find the container with id f6e6b962d50a718098bf2af96f094ad7b26723de576d88101109078480742cdc Jun 02 01:41:24.164445 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:24.164426 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tglfq" Jun 02 01:41:24.169849 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:24.169829 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" Jun 02 01:41:24.169996 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:24.169973 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae3e1499_5744_4270_b94d_f292d30bd1b2.slice/crio-ee8d4566116242a325d553426733b3d8ed4a2afa9676ef68bf14c0454651d5b1 WatchSource:0}: Error finding container ee8d4566116242a325d553426733b3d8ed4a2afa9676ef68bf14c0454651d5b1: Status 404 returned error can't find the container with id ee8d4566116242a325d553426733b3d8ed4a2afa9676ef68bf14c0454651d5b1 Jun 02 01:41:24.176738 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:24.176715 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa0d7d15_d800_4698_a39f_763402e48110.slice/crio-e67346e74ee491f783b9364b367eee6dd5f5b4189d3d675346abcb5f968f53f0 WatchSource:0}: Error finding container e67346e74ee491f783b9364b367eee6dd5f5b4189d3d675346abcb5f968f53f0: Status 404 returned error can't find the container with id e67346e74ee491f783b9364b367eee6dd5f5b4189d3d675346abcb5f968f53f0 Jun 02 01:41:24.177357 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:24.177336 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nm89v" Jun 02 01:41:24.183479 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:24.183459 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7drhb" Jun 02 01:41:24.184345 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:24.184306 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c45e264_330c_4438_bdc1_817a0a8b1b77.slice/crio-e3338d0b79fef2af95c2b98768c3bb170aa691075368715501616d5991a7d229 WatchSource:0}: Error finding container e3338d0b79fef2af95c2b98768c3bb170aa691075368715501616d5991a7d229: Status 404 returned error can't find the container with id e3338d0b79fef2af95c2b98768c3bb170aa691075368715501616d5991a7d229 Jun 02 01:41:24.189225 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:24.189203 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba65c8da_80cd_4fda_9746_dd851a3a06be.slice/crio-5d5137a2eff5a2d64644fa7985c728cad324d8f1ceb59918205e88fcacf86e39 WatchSource:0}: Error finding container 5d5137a2eff5a2d64644fa7985c728cad324d8f1ceb59918205e88fcacf86e39: Status 404 returned error can't find the container with id 5d5137a2eff5a2d64644fa7985c728cad324d8f1ceb59918205e88fcacf86e39 Jun 02 01:41:24.189949 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:24.189931 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6dnbd" Jun 02 01:41:24.193799 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:24.193784 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bmd65" Jun 02 01:41:24.195660 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:24.195640 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f82072c_4fa6_41c2_91ce_6c7cd16a0840.slice/crio-db1d3f8d566acaa04c68d3178b772448c1be19d6f570238427eabaf786c85738 WatchSource:0}: Error finding container db1d3f8d566acaa04c68d3178b772448c1be19d6f570238427eabaf786c85738: Status 404 returned error can't find the container with id db1d3f8d566acaa04c68d3178b772448c1be19d6f570238427eabaf786c85738 Jun 02 01:41:24.200077 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:24.200058 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab3aef63_c4c0_4b01_b26b_53ca0e39e7bb.slice/crio-f4af9a371b91fe2ccd382860c89aaffb24fdb78c3f9efcd5de482fd98e97d1f4 WatchSource:0}: Error finding container f4af9a371b91fe2ccd382860c89aaffb24fdb78c3f9efcd5de482fd98e97d1f4: Status 404 returned error can't find the container with id f4af9a371b91fe2ccd382860c89aaffb24fdb78c3f9efcd5de482fd98e97d1f4 Jun 02 01:41:24.409440 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:24.409345 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3542744-21d2-4162-a2f9-4a474d18dc19-metrics-certs\") pod \"network-metrics-daemon-smqwh\" (UID: \"f3542744-21d2-4162-a2f9-4a474d18dc19\") " pod="openshift-multus/network-metrics-daemon-smqwh" Jun 02 01:41:24.409592 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:24.409518 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jun 02 01:41:24.409592 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:24.409584 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3542744-21d2-4162-a2f9-4a474d18dc19-metrics-certs podName:f3542744-21d2-4162-a2f9-4a474d18dc19 nodeName:}" failed. No retries permitted until 2026-06-02 01:41:25.409565451 +0000 UTC m=+3.055372071 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f3542744-21d2-4162-a2f9-4a474d18dc19-metrics-certs") pod "network-metrics-daemon-smqwh" (UID: "f3542744-21d2-4162-a2f9-4a474d18dc19") : object "openshift-multus"/"metrics-daemon-secret" not registered Jun 02 01:41:24.510480 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:24.510433 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6f7hb\" (UniqueName: \"kubernetes.io/projected/7db70fbd-3197-47ed-a00b-d6b4fbb4c8a8-kube-api-access-6f7hb\") pod \"network-check-target-gt4w8\" (UID: \"7db70fbd-3197-47ed-a00b-d6b4fbb4c8a8\") " pod="openshift-network-diagnostics/network-check-target-gt4w8" Jun 02 01:41:24.510664 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:24.510651 2571 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jun 02 01:41:24.510722 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:24.510675 2571 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jun 02 01:41:24.510722 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:24.510688 2571 projected.go:194] Error preparing data for projected volume kube-api-access-6f7hb for pod openshift-network-diagnostics/network-check-target-gt4w8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 02 01:41:24.510833 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:24.510747 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7db70fbd-3197-47ed-a00b-d6b4fbb4c8a8-kube-api-access-6f7hb podName:7db70fbd-3197-47ed-a00b-d6b4fbb4c8a8 nodeName:}" failed. No retries permitted until 2026-06-02 01:41:25.510729264 +0000 UTC m=+3.156535890 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-6f7hb" (UniqueName: "kubernetes.io/projected/7db70fbd-3197-47ed-a00b-d6b4fbb4c8a8-kube-api-access-6f7hb") pod "network-check-target-gt4w8" (UID: "7db70fbd-3197-47ed-a00b-d6b4fbb4c8a8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 02 01:41:24.589674 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:24.589634 2571 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jun 02 01:41:24.741723 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:24.741422 2571 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jun 02 01:41:24.831486 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:24.831450 2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-01 01:36:23 +0000 UTC" deadline="2028-02-09 20:59:34.871586398 +0000 UTC" Jun 02 01:41:24.831486 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:24.831482 2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14827h18m10.040107479s" Jun 02 01:41:24.953332 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:24.953295 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nm89v" event={"ID":"5c45e264-330c-4438-bdc1-817a0a8b1b77","Type":"ContainerStarted","Data":"e3338d0b79fef2af95c2b98768c3bb170aa691075368715501616d5991a7d229"} Jun 02 01:41:24.959744 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:24.959708 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" event={"ID":"aa0d7d15-d800-4698-a39f-763402e48110","Type":"ContainerStarted","Data":"e67346e74ee491f783b9364b367eee6dd5f5b4189d3d675346abcb5f968f53f0"} Jun 02 01:41:24.962453 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:24.962423 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tglfq" event={"ID":"ae3e1499-5744-4270-b94d-f292d30bd1b2","Type":"ContainerStarted","Data":"ee8d4566116242a325d553426733b3d8ed4a2afa9676ef68bf14c0454651d5b1"} Jun 02 01:41:24.972875 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:24.972842 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-thlx8" event={"ID":"4b78335d-1f32-4d95-adda-efac2d74e4e1","Type":"ContainerStarted","Data":"7ab1af8481a0b35f003c4cde449ba3ee4e3e077aeaddf71707d449d2b6d6688c"} Jun 02 01:41:24.989828 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:24.989792 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7drhb" event={"ID":"ba65c8da-80cd-4fda-9746-dd851a3a06be","Type":"ContainerStarted","Data":"5d5137a2eff5a2d64644fa7985c728cad324d8f1ceb59918205e88fcacf86e39"} Jun 02 01:41:24.997868 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:24.997786 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6rzvj" event={"ID":"1fc82c63-2678-486d-9393-c3ea7f27abce","Type":"ContainerStarted","Data":"f6e6b962d50a718098bf2af96f094ad7b26723de576d88101109078480742cdc"} Jun 02 01:41:25.005471 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:25.005430 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zf8v" event={"ID":"5cffe19e-fcbd-46bb-9db3-2617a77f95f6","Type":"ContainerStarted","Data":"1169e7aef6aab904d368b443a10e6b557130b587366574843139b84570c5bc4e"} Jun 02 01:41:25.016488 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:25.016427 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bmd65" event={"ID":"ab3aef63-c4c0-4b01-b26b-53ca0e39e7bb","Type":"ContainerStarted","Data":"f4af9a371b91fe2ccd382860c89aaffb24fdb78c3f9efcd5de482fd98e97d1f4"} Jun 02 01:41:25.022394 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:25.022341 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6dnbd" event={"ID":"8f82072c-4fa6-41c2-91ce-6c7cd16a0840","Type":"ContainerStarted","Data":"db1d3f8d566acaa04c68d3178b772448c1be19d6f570238427eabaf786c85738"} Jun 02 01:41:25.416858 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:25.416234 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3542744-21d2-4162-a2f9-4a474d18dc19-metrics-certs\") pod \"network-metrics-daemon-smqwh\" (UID: \"f3542744-21d2-4162-a2f9-4a474d18dc19\") " pod="openshift-multus/network-metrics-daemon-smqwh" Jun 02 01:41:25.416858 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:25.416388 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jun 02 01:41:25.416858 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:25.416462 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3542744-21d2-4162-a2f9-4a474d18dc19-metrics-certs podName:f3542744-21d2-4162-a2f9-4a474d18dc19 nodeName:}" failed. No retries permitted until 2026-06-02 01:41:27.416442806 +0000 UTC m=+5.062249430 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f3542744-21d2-4162-a2f9-4a474d18dc19-metrics-certs") pod "network-metrics-daemon-smqwh" (UID: "f3542744-21d2-4162-a2f9-4a474d18dc19") : object "openshift-multus"/"metrics-daemon-secret" not registered Jun 02 01:41:25.517409 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:25.517372 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6f7hb\" (UniqueName: \"kubernetes.io/projected/7db70fbd-3197-47ed-a00b-d6b4fbb4c8a8-kube-api-access-6f7hb\") pod \"network-check-target-gt4w8\" (UID: \"7db70fbd-3197-47ed-a00b-d6b4fbb4c8a8\") " pod="openshift-network-diagnostics/network-check-target-gt4w8" Jun 02 01:41:25.517592 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:25.517543 2571 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jun 02 01:41:25.517592 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:25.517565 2571 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jun 02 01:41:25.517592 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:25.517580 2571 projected.go:194] Error preparing data for projected volume kube-api-access-6f7hb for pod openshift-network-diagnostics/network-check-target-gt4w8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 02 01:41:25.517743 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:25.517637 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7db70fbd-3197-47ed-a00b-d6b4fbb4c8a8-kube-api-access-6f7hb podName:7db70fbd-3197-47ed-a00b-d6b4fbb4c8a8 nodeName:}" failed. No retries permitted until 2026-06-02 01:41:27.517619773 +0000 UTC m=+5.163426399 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-6f7hb" (UniqueName: "kubernetes.io/projected/7db70fbd-3197-47ed-a00b-d6b4fbb4c8a8-kube-api-access-6f7hb") pod "network-check-target-gt4w8" (UID: "7db70fbd-3197-47ed-a00b-d6b4fbb4c8a8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 02 01:41:25.643395 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:25.643045 2571 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jun 02 01:41:25.832644 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:25.832542 2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-01 01:36:23 +0000 UTC" deadline="2028-02-10 14:05:15.74655941 +0000 UTC" Jun 02 01:41:25.832644 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:25.832580 2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14844h23m49.913982668s" Jun 02 01:41:25.924074 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:25.924044 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gt4w8" Jun 02 01:41:25.924276 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:25.924207 2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gt4w8" podUID="7db70fbd-3197-47ed-a00b-d6b4fbb4c8a8" Jun 02 01:41:25.924661 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:25.924641 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-smqwh" Jun 02 01:41:25.924772 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:25.924752 2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-smqwh" podUID="f3542744-21d2-4162-a2f9-4a474d18dc19" Jun 02 01:41:27.433505 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:27.433471 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3542744-21d2-4162-a2f9-4a474d18dc19-metrics-certs\") pod \"network-metrics-daemon-smqwh\" (UID: \"f3542744-21d2-4162-a2f9-4a474d18dc19\") " pod="openshift-multus/network-metrics-daemon-smqwh" Jun 02 01:41:27.433988 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:27.433677 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jun 02 01:41:27.433988 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:27.433740 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3542744-21d2-4162-a2f9-4a474d18dc19-metrics-certs podName:f3542744-21d2-4162-a2f9-4a474d18dc19 nodeName:}" failed. No retries permitted until 2026-06-02 01:41:31.43372241 +0000 UTC m=+9.079529047 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f3542744-21d2-4162-a2f9-4a474d18dc19-metrics-certs") pod "network-metrics-daemon-smqwh" (UID: "f3542744-21d2-4162-a2f9-4a474d18dc19") : object "openshift-multus"/"metrics-daemon-secret" not registered Jun 02 01:41:27.534736 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:27.534699 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6f7hb\" (UniqueName: \"kubernetes.io/projected/7db70fbd-3197-47ed-a00b-d6b4fbb4c8a8-kube-api-access-6f7hb\") pod \"network-check-target-gt4w8\" (UID: \"7db70fbd-3197-47ed-a00b-d6b4fbb4c8a8\") " pod="openshift-network-diagnostics/network-check-target-gt4w8" Jun 02 01:41:27.534996 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:27.534903 2571 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jun 02 01:41:27.534996 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:27.534922 2571 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jun 02 01:41:27.534996 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:27.534935 2571 projected.go:194] Error preparing data for projected volume kube-api-access-6f7hb for pod openshift-network-diagnostics/network-check-target-gt4w8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 02 01:41:27.534996 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:27.534997 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7db70fbd-3197-47ed-a00b-d6b4fbb4c8a8-kube-api-access-6f7hb podName:7db70fbd-3197-47ed-a00b-d6b4fbb4c8a8 nodeName:}" failed. No retries permitted until 2026-06-02 01:41:31.534978761 +0000 UTC m=+9.180785394 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-6f7hb" (UniqueName: "kubernetes.io/projected/7db70fbd-3197-47ed-a00b-d6b4fbb4c8a8-kube-api-access-6f7hb") pod "network-check-target-gt4w8" (UID: "7db70fbd-3197-47ed-a00b-d6b4fbb4c8a8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 02 01:41:27.924287 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:27.924209 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gt4w8" Jun 02 01:41:27.924447 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:27.924325 2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gt4w8" podUID="7db70fbd-3197-47ed-a00b-d6b4fbb4c8a8" Jun 02 01:41:27.924447 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:27.924209 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-smqwh" Jun 02 01:41:27.924447 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:27.924419 2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-smqwh" podUID="f3542744-21d2-4162-a2f9-4a474d18dc19" Jun 02 01:41:29.924444 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:29.924290 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gt4w8" Jun 02 01:41:29.924444 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:29.924429 2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gt4w8" podUID="7db70fbd-3197-47ed-a00b-d6b4fbb4c8a8" Jun 02 01:41:29.924961 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:29.924290 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-smqwh" Jun 02 01:41:29.924961 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:29.924884 2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-smqwh" podUID="f3542744-21d2-4162-a2f9-4a474d18dc19" Jun 02 01:41:31.468400 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:31.468172 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3542744-21d2-4162-a2f9-4a474d18dc19-metrics-certs\") pod \"network-metrics-daemon-smqwh\" (UID: \"f3542744-21d2-4162-a2f9-4a474d18dc19\") " pod="openshift-multus/network-metrics-daemon-smqwh" Jun 02 01:41:31.468400 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:31.468361 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jun 02 01:41:31.468909 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:31.468435 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3542744-21d2-4162-a2f9-4a474d18dc19-metrics-certs podName:f3542744-21d2-4162-a2f9-4a474d18dc19 nodeName:}" failed. No retries permitted until 2026-06-02 01:41:39.468414697 +0000 UTC m=+17.114221326 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f3542744-21d2-4162-a2f9-4a474d18dc19-metrics-certs") pod "network-metrics-daemon-smqwh" (UID: "f3542744-21d2-4162-a2f9-4a474d18dc19") : object "openshift-multus"/"metrics-daemon-secret" not registered Jun 02 01:41:31.568753 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:31.568618 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6f7hb\" (UniqueName: \"kubernetes.io/projected/7db70fbd-3197-47ed-a00b-d6b4fbb4c8a8-kube-api-access-6f7hb\") pod \"network-check-target-gt4w8\" (UID: \"7db70fbd-3197-47ed-a00b-d6b4fbb4c8a8\") " pod="openshift-network-diagnostics/network-check-target-gt4w8" Jun 02 01:41:31.568955 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:31.568781 2571 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jun 02 01:41:31.568955 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:31.568801 2571 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jun 02 01:41:31.568955 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:31.568813 2571 projected.go:194] Error preparing data for projected volume kube-api-access-6f7hb for pod openshift-network-diagnostics/network-check-target-gt4w8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 02 01:41:31.568955 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:31.568880 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7db70fbd-3197-47ed-a00b-d6b4fbb4c8a8-kube-api-access-6f7hb podName:7db70fbd-3197-47ed-a00b-d6b4fbb4c8a8 nodeName:}" failed. No retries permitted until 2026-06-02 01:41:39.568860384 +0000 UTC m=+17.214667010 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-6f7hb" (UniqueName: "kubernetes.io/projected/7db70fbd-3197-47ed-a00b-d6b4fbb4c8a8-kube-api-access-6f7hb") pod "network-check-target-gt4w8" (UID: "7db70fbd-3197-47ed-a00b-d6b4fbb4c8a8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 02 01:41:31.923596 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:31.923501 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gt4w8" Jun 02 01:41:31.923596 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:31.923506 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-smqwh" Jun 02 01:41:31.923829 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:31.923645 2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gt4w8" podUID="7db70fbd-3197-47ed-a00b-d6b4fbb4c8a8" Jun 02 01:41:31.923948 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:31.923915 2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-smqwh" podUID="f3542744-21d2-4162-a2f9-4a474d18dc19" Jun 02 01:41:33.923991 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:33.923950 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-smqwh" Jun 02 01:41:33.923991 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:33.923985 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gt4w8" Jun 02 01:41:33.924574 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:33.924088 2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-smqwh" podUID="f3542744-21d2-4162-a2f9-4a474d18dc19" Jun 02 01:41:33.924574 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:33.924223 2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gt4w8" podUID="7db70fbd-3197-47ed-a00b-d6b4fbb4c8a8" Jun 02 01:41:35.923968 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:35.923928 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gt4w8" Jun 02 01:41:35.924529 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:35.924055 2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gt4w8" podUID="7db70fbd-3197-47ed-a00b-d6b4fbb4c8a8" Jun 02 01:41:35.924529 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:35.924107 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-smqwh" Jun 02 01:41:35.924529 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:35.924231 2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-smqwh" podUID="f3542744-21d2-4162-a2f9-4a474d18dc19" Jun 02 01:41:37.923821 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:37.923628 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-smqwh" Jun 02 01:41:37.924241 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:37.923629 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gt4w8" Jun 02 01:41:37.924241 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:37.923924 2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-smqwh" podUID="f3542744-21d2-4162-a2f9-4a474d18dc19" Jun 02 01:41:37.924241 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:37.923993 2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gt4w8" podUID="7db70fbd-3197-47ed-a00b-d6b4fbb4c8a8" Jun 02 01:41:39.525382 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:39.525344 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3542744-21d2-4162-a2f9-4a474d18dc19-metrics-certs\") pod \"network-metrics-daemon-smqwh\" (UID: \"f3542744-21d2-4162-a2f9-4a474d18dc19\") " pod="openshift-multus/network-metrics-daemon-smqwh" Jun 02 01:41:39.525842 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:39.525462 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jun 02 01:41:39.525842 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:39.525529 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3542744-21d2-4162-a2f9-4a474d18dc19-metrics-certs podName:f3542744-21d2-4162-a2f9-4a474d18dc19 nodeName:}" failed. No retries permitted until 2026-06-02 01:41:55.525509585 +0000 UTC m=+33.171316209 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f3542744-21d2-4162-a2f9-4a474d18dc19-metrics-certs") pod "network-metrics-daemon-smqwh" (UID: "f3542744-21d2-4162-a2f9-4a474d18dc19") : object "openshift-multus"/"metrics-daemon-secret" not registered Jun 02 01:41:39.626672 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:39.626590 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6f7hb\" (UniqueName: \"kubernetes.io/projected/7db70fbd-3197-47ed-a00b-d6b4fbb4c8a8-kube-api-access-6f7hb\") pod \"network-check-target-gt4w8\" (UID: \"7db70fbd-3197-47ed-a00b-d6b4fbb4c8a8\") " pod="openshift-network-diagnostics/network-check-target-gt4w8" Jun 02 01:41:39.626854 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:39.626754 2571 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jun 02 01:41:39.626854 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:39.626786 2571 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jun 02 01:41:39.626854 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:39.626799 2571 projected.go:194] Error preparing data for projected volume kube-api-access-6f7hb for pod openshift-network-diagnostics/network-check-target-gt4w8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 02 01:41:39.627016 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:39.626863 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7db70fbd-3197-47ed-a00b-d6b4fbb4c8a8-kube-api-access-6f7hb podName:7db70fbd-3197-47ed-a00b-d6b4fbb4c8a8 nodeName:}" failed. No retries permitted until 2026-06-02 01:41:55.62684609 +0000 UTC m=+33.272652725 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-6f7hb" (UniqueName: "kubernetes.io/projected/7db70fbd-3197-47ed-a00b-d6b4fbb4c8a8-kube-api-access-6f7hb") pod "network-check-target-gt4w8" (UID: "7db70fbd-3197-47ed-a00b-d6b4fbb4c8a8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 02 01:41:39.923950 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:39.923913 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gt4w8" Jun 02 01:41:39.924102 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:39.924029 2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gt4w8" podUID="7db70fbd-3197-47ed-a00b-d6b4fbb4c8a8" Jun 02 01:41:39.924102 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:39.924086 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-smqwh" Jun 02 01:41:39.924235 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:39.924197 2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-smqwh" podUID="f3542744-21d2-4162-a2f9-4a474d18dc19" Jun 02 01:41:41.924191 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:41.924150 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-smqwh" Jun 02 01:41:41.924653 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:41.924150 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gt4w8" Jun 02 01:41:41.924653 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:41.924291 2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-smqwh" podUID="f3542744-21d2-4162-a2f9-4a474d18dc19" Jun 02 01:41:41.924653 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:41.924397 2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gt4w8" podUID="7db70fbd-3197-47ed-a00b-d6b4fbb4c8a8" Jun 02 01:41:43.923560 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:43.923379 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gt4w8" Jun 02 01:41:43.924115 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:43.923634 2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gt4w8" podUID="7db70fbd-3197-47ed-a00b-d6b4fbb4c8a8" Jun 02 01:41:43.924115 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:43.923939 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-smqwh" Jun 02 01:41:43.924115 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:43.924010 2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-smqwh" podUID="f3542744-21d2-4162-a2f9-4a474d18dc19" Jun 02 01:41:44.064446 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:44.064414 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-117.ec2.internal" event={"ID":"dddc001fa74725ad60cc8708aa217f2d","Type":"ContainerStarted","Data":"ca15981633aa025cd643c7c477580ba698f6eac8049bd0d1bf2daa90b9032021"} Jun 02 01:41:44.068630 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:44.068602 2571 generic.go:358] "Generic (PLEG): container finished" podID="ee07ddb5f970db1345c2560076c9a2b8" containerID="81268d923adacf9e8895bd0952a31a108bddbf76c3977a7d44243bf9f59ee164" exitCode=0 Jun 02 01:41:44.068738 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:44.068676 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-117.ec2.internal" event={"ID":"ee07ddb5f970db1345c2560076c9a2b8","Type":"ContainerDied","Data":"81268d923adacf9e8895bd0952a31a108bddbf76c3977a7d44243bf9f59ee164"} Jun 02 01:41:44.068826 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:44.068813 2571 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-117.ec2.internal" Jun 02 01:41:44.069889 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:44.069868 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bmd65" event={"ID":"ab3aef63-c4c0-4b01-b26b-53ca0e39e7bb","Type":"ContainerStarted","Data":"16e79f4996711f0d7e57601b6e7b19f1e7e8b84b584773c3973229133104d8c7"} Jun 02 01:41:44.070994 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:44.070977 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nm89v" event={"ID":"5c45e264-330c-4438-bdc1-817a0a8b1b77","Type":"ContainerStarted","Data":"d11abb780824928202b0fb40619466289a7df1cce3ee6d611d17238fe9777943"} Jun 02 01:41:44.072854 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:44.072834 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" event={"ID":"aa0d7d15-d800-4698-a39f-763402e48110","Type":"ContainerStarted","Data":"1f8d356d45392ff2664c928564eb3a4cfa66e255262aeffbfd7120e3102c49e7"} Jun 02 01:41:44.072958 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:44.072859 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" event={"ID":"aa0d7d15-d800-4698-a39f-763402e48110","Type":"ContainerStarted","Data":"a38a5c7d412d8cc8e46230212afd47bef0ff40c153d2e434aff5de9efbb674af"} Jun 02 01:41:44.072958 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:44.072874 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" event={"ID":"aa0d7d15-d800-4698-a39f-763402e48110","Type":"ContainerStarted","Data":"843d0195fe96cad37a7e1039f6149a2b51bb9bef69a5cafa9baf0587e19ab949"} Jun 02 01:41:44.072958 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:44.072887 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" event={"ID":"aa0d7d15-d800-4698-a39f-763402e48110","Type":"ContainerStarted","Data":"bd8d8082a7634fa843c8552a9a78b011093718315d1614da034b938c371e5b94"} Jun 02 01:41:44.076201 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:44.076179 2571 generic.go:358] "Generic (PLEG): container finished" podID="ae3e1499-5744-4270-b94d-f292d30bd1b2" containerID="b98a8ede24217982ffec776459e2f096ad654cb73224e15bb186e2719bdb939b" exitCode=0 Jun 02 01:41:44.076293 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:44.076264 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tglfq" event={"ID":"ae3e1499-5744-4270-b94d-f292d30bd1b2","Type":"ContainerDied","Data":"b98a8ede24217982ffec776459e2f096ad654cb73224e15bb186e2719bdb939b"} Jun 02 01:41:44.077611 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:44.077571 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-thlx8" event={"ID":"4b78335d-1f32-4d95-adda-efac2d74e4e1","Type":"ContainerStarted","Data":"a1c702a09564f8d4b9ac83e1bb65a59539b71728f90e7f4696b6d3e54a694e05"} Jun 02 01:41:44.077857 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:44.077696 2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-117.ec2.internal" podStartSLOduration=21.077684011 podStartE2EDuration="21.077684011s" podCreationTimestamp="2026-06-02 01:41:23 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 01:41:44.077288009 +0000 UTC m=+21.723094652" watchObservedRunningTime="2026-06-02 01:41:44.077684011 +0000 UTC m=+21.723490653" Jun 02 01:41:44.078923 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:44.078906 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7drhb" event={"ID":"ba65c8da-80cd-4fda-9746-dd851a3a06be","Type":"ContainerStarted","Data":"2bb88807a125553887f9ace2fb50d1dad9fbdb3abd527cde7db0ea83acec228f"} Jun 02 01:41:44.080015 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:44.079996 2571 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jun 02 01:41:44.080212 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:44.080192 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6rzvj" event={"ID":"1fc82c63-2678-486d-9393-c3ea7f27abce","Type":"ContainerStarted","Data":"c65720c3125c074a035871bb3d84fc301e17f81681a2dd48cbded9eb7b33d440"} Jun 02 01:41:44.080784 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:44.080766 2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-117.ec2.internal"] Jun 02 01:41:44.081523 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:44.081505 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zf8v" event={"ID":"5cffe19e-fcbd-46bb-9db3-2617a77f95f6","Type":"ContainerStarted","Data":"ca96bd4b4c1cdd0198adf548c0988180ca51d881f3f3cfaed6a7d406b3885536"} Jun 02 01:41:44.102890 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:44.102849 2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-bmd65" podStartSLOduration=2.262032805 podStartE2EDuration="21.102834052s" podCreationTimestamp="2026-06-02 01:41:23 +0000 UTC" firstStartedPulling="2026-06-02 01:41:24.201453003 +0000 UTC m=+1.847259623" lastFinishedPulling="2026-06-02 01:41:43.042254242 +0000 UTC m=+20.688060870" observedRunningTime="2026-06-02 01:41:44.102715462 +0000 UTC m=+21.748522105" watchObservedRunningTime="2026-06-02 01:41:44.102834052 +0000 UTC m=+21.748640688" Jun 02 01:41:44.103354 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:44.103314 2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-nm89v" podStartSLOduration=2.218852813 podStartE2EDuration="21.103305879s" podCreationTimestamp="2026-06-02 01:41:23 +0000 UTC" firstStartedPulling="2026-06-02 01:41:24.186491429 +0000 UTC m=+1.832298049" lastFinishedPulling="2026-06-02 01:41:43.070944484 +0000 UTC m=+20.716751115" observedRunningTime="2026-06-02 01:41:44.090208761 +0000 UTC m=+21.736015403" watchObservedRunningTime="2026-06-02 01:41:44.103305879 +0000 UTC m=+21.749112564" Jun 02 01:41:44.140147 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:44.140065 2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-7drhb" podStartSLOduration=2.0544438 podStartE2EDuration="21.140046586s" podCreationTimestamp="2026-06-02 01:41:23 +0000 UTC" firstStartedPulling="2026-06-02 01:41:24.19141768 +0000 UTC m=+1.837224314" lastFinishedPulling="2026-06-02 01:41:43.277020479 +0000 UTC m=+20.922827100" observedRunningTime="2026-06-02 01:41:44.139867214 +0000 UTC m=+21.785673858" watchObservedRunningTime="2026-06-02 01:41:44.140046586 +0000 UTC m=+21.785853230" Jun 02 01:41:44.169029 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:44.168994 2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-6rzvj" podStartSLOduration=2.279547651 podStartE2EDuration="21.168981013s" podCreationTimestamp="2026-06-02 01:41:23 +0000 UTC" firstStartedPulling="2026-06-02 01:41:24.15282228 +0000 UTC m=+1.798628902" lastFinishedPulling="2026-06-02 01:41:43.042255627 +0000 UTC m=+20.688062264" observedRunningTime="2026-06-02 01:41:44.168806958 +0000 UTC m=+21.814613600" watchObservedRunningTime="2026-06-02 01:41:44.168981013 +0000 UTC m=+21.814787655" Jun 02 01:41:44.183704 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:44.183669 2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-thlx8" podStartSLOduration=3.025590205 podStartE2EDuration="22.183654634s" podCreationTimestamp="2026-06-02 01:41:22 +0000 UTC" firstStartedPulling="2026-06-02 01:41:24.118942594 +0000 UTC m=+1.764749214" lastFinishedPulling="2026-06-02 01:41:43.277007017 +0000 UTC m=+20.922813643" observedRunningTime="2026-06-02 01:41:44.183526402 +0000 UTC m=+21.829333043" watchObservedRunningTime="2026-06-02 01:41:44.183654634 +0000 UTC m=+21.829461275" Jun 02 01:41:45.080550 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:45.080517 2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-bmd65" Jun 02 01:41:45.081194 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:45.081169 2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-bmd65" Jun 02 01:41:45.086747 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:45.086718 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-117.ec2.internal" event={"ID":"ee07ddb5f970db1345c2560076c9a2b8","Type":"ContainerStarted","Data":"678e180d5f4bc63ee13c61afe41b44ac7dc944107ce72e44dff82f981f0f1bc2"} Jun 02 01:41:45.088371 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:45.088330 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6dnbd" event={"ID":"8f82072c-4fa6-41c2-91ce-6c7cd16a0840","Type":"ContainerStarted","Data":"27b61492e7a11ed0b8e68eb50df4b8263503ac2448286008ce180e3a28964dfd"} Jun 02 01:41:45.091510 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:45.091188 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" event={"ID":"aa0d7d15-d800-4698-a39f-763402e48110","Type":"ContainerStarted","Data":"8ef3d599325459f0e4bc2425eb784358a0bc2c3b450ba131d31a3c34534dc771"} Jun 02 01:41:45.091510 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:45.091219 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" event={"ID":"aa0d7d15-d800-4698-a39f-763402e48110","Type":"ContainerStarted","Data":"aaaf65dfa930a3b7cc27df45c4a7dc071ced009e31c50a2d5809632b2840f814"} Jun 02 01:41:45.110655 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:45.110565 2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-6dnbd" podStartSLOduration=3.031024826 podStartE2EDuration="22.110547334s" podCreationTimestamp="2026-06-02 01:41:23 +0000 UTC" firstStartedPulling="2026-06-02 01:41:24.197212934 +0000 UTC m=+1.843019554" lastFinishedPulling="2026-06-02 01:41:43.276735428 +0000 UTC m=+20.922542062" observedRunningTime="2026-06-02 01:41:45.109833699 +0000 UTC m=+22.755640343" watchObservedRunningTime="2026-06-02 01:41:45.110547334 +0000 UTC m=+22.756353977" Jun 02 01:41:45.124669 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:45.124623 2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-117.ec2.internal" podStartSLOduration=1.124605781 podStartE2EDuration="1.124605781s" podCreationTimestamp="2026-06-02 01:41:44 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 01:41:45.12409165 +0000 UTC m=+22.769898292" watchObservedRunningTime="2026-06-02 01:41:45.124605781 +0000 UTC m=+22.770412424" Jun 02 01:41:45.184807 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:45.184776 2571 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jun 02 01:41:45.874337 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:45.874211 2571 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-06-02T01:41:45.184800126Z","UUID":"214e646c-3c1a-4c66-ac3d-ca8a53972a03","Handler":null,"Name":"","Endpoint":""} Jun 02 01:41:45.876406 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:45.876364 2571 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jun 02 01:41:45.876406 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:45.876400 2571 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jun 02 01:41:45.924016 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:45.923981 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gt4w8" Jun 02 01:41:45.924214 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:45.923981 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-smqwh" Jun 02 01:41:45.924214 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:45.924114 2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gt4w8" podUID="7db70fbd-3197-47ed-a00b-d6b4fbb4c8a8" Jun 02 01:41:45.924214 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:45.924174 2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-smqwh" podUID="f3542744-21d2-4162-a2f9-4a474d18dc19" Jun 02 01:41:46.094977 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:46.094914 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zf8v" event={"ID":"5cffe19e-fcbd-46bb-9db3-2617a77f95f6","Type":"ContainerStarted","Data":"87d8bb30de66f3a92c8763ce2769f151344318bc2d2469d1b91c1ba1ebf68fce"} Jun 02 01:41:46.094977 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:46.094969 2571 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jun 02 01:41:47.100308 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:47.100252 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" event={"ID":"aa0d7d15-d800-4698-a39f-763402e48110","Type":"ContainerStarted","Data":"a7ba4566b32003e437442ab12701e026443073ce91887635be8dc97255885bf2"} Jun 02 01:41:47.102166 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:47.102120 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zf8v" event={"ID":"5cffe19e-fcbd-46bb-9db3-2617a77f95f6","Type":"ContainerStarted","Data":"d68897a774d6506dee86749fac8b894f20f2e44951cd76880dad2c1b3b24b701"} Jun 02 01:41:47.119905 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:47.119862 2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zf8v" podStartSLOduration=2.876738591 podStartE2EDuration="25.119848336s" podCreationTimestamp="2026-06-02 01:41:22 +0000 UTC" firstStartedPulling="2026-06-02 01:41:24.136124028 +0000 UTC m=+1.781930652" lastFinishedPulling="2026-06-02 01:41:46.379233766 +0000 UTC m=+24.025040397" observedRunningTime="2026-06-02 01:41:47.119762474 +0000 UTC m=+24.765569116" watchObservedRunningTime="2026-06-02 01:41:47.119848336 +0000 UTC m=+24.765654978" Jun 02 01:41:47.924193 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:47.924158 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gt4w8" Jun 02 01:41:47.924193 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:47.924203 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-smqwh" Jun 02 01:41:47.924445 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:47.924279 2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gt4w8" podUID="7db70fbd-3197-47ed-a00b-d6b4fbb4c8a8" Jun 02 01:41:47.924445 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:47.924400 2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-smqwh" podUID="f3542744-21d2-4162-a2f9-4a474d18dc19" Jun 02 01:41:49.108154 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:49.107891 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" event={"ID":"aa0d7d15-d800-4698-a39f-763402e48110","Type":"ContainerStarted","Data":"bb55b271d3b0ed5609bde10c2a80e9c15b9e1e56148827a46a0594dc315fae19"} Jun 02 01:41:49.108658 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:49.108164 2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" Jun 02 01:41:49.108658 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:49.108217 2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" Jun 02 01:41:49.109595 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:49.109558 2571 generic.go:358] "Generic (PLEG): container finished" podID="ae3e1499-5744-4270-b94d-f292d30bd1b2" containerID="0fd71b869e41562781f4aef94d48d464a929d43fb92c1c10927b994d697ebfe7" exitCode=0 Jun 02 01:41:49.109713 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:49.109608 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tglfq" event={"ID":"ae3e1499-5744-4270-b94d-f292d30bd1b2","Type":"ContainerDied","Data":"0fd71b869e41562781f4aef94d48d464a929d43fb92c1c10927b994d697ebfe7"} Jun 02 01:41:49.123713 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:49.123692 2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" Jun 02 01:41:49.134652 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:49.134608 2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" podStartSLOduration=6.702932585 podStartE2EDuration="26.134596582s" podCreationTimestamp="2026-06-02 01:41:23 +0000 UTC" firstStartedPulling="2026-06-02 01:41:24.178026421 +0000 UTC m=+1.823833041" lastFinishedPulling="2026-06-02 01:41:43.609690415 +0000 UTC m=+21.255497038" observedRunningTime="2026-06-02 01:41:49.134003894 +0000 UTC m=+26.779810530" watchObservedRunningTime="2026-06-02 01:41:49.134596582 +0000 UTC m=+26.780403224" Jun 02 01:41:49.926417 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:49.926365 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gt4w8" Jun 02 01:41:49.926702 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:49.926553 2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gt4w8" podUID="7db70fbd-3197-47ed-a00b-d6b4fbb4c8a8" Jun 02 01:41:49.927281 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:49.927249 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-smqwh" Jun 02 01:41:49.927403 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:49.927371 2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-smqwh" podUID="f3542744-21d2-4162-a2f9-4a474d18dc19" Jun 02 01:41:50.112188 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:50.112160 2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" Jun 02 01:41:50.126719 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:50.126695 2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" Jun 02 01:41:50.148964 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:50.148931 2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-bmd65" Jun 02 01:41:50.149163 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:50.149067 2571 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jun 02 01:41:50.149556 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:50.149536 2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-bmd65" Jun 02 01:41:50.571361 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:50.571238 2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-smqwh"] Jun 02 01:41:50.571361 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:50.571354 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-smqwh" Jun 02 01:41:50.571575 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:50.571458 2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-smqwh" podUID="f3542744-21d2-4162-a2f9-4a474d18dc19" Jun 02 01:41:50.571878 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:50.571858 2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gt4w8"] Jun 02 01:41:50.571960 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:50.571932 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gt4w8" Jun 02 01:41:50.572120 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:50.572004 2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gt4w8" podUID="7db70fbd-3197-47ed-a00b-d6b4fbb4c8a8" Jun 02 01:41:51.115394 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:51.115362 2571 generic.go:358] "Generic (PLEG): container finished" podID="ae3e1499-5744-4270-b94d-f292d30bd1b2" containerID="8f21c0088416cfbe0bc5bd81fffdaf9ef1c9b938ba77ef058ac9c898243f6097" exitCode=0 Jun 02 01:41:51.115780 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:51.115443 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tglfq" event={"ID":"ae3e1499-5744-4270-b94d-f292d30bd1b2","Type":"ContainerDied","Data":"8f21c0088416cfbe0bc5bd81fffdaf9ef1c9b938ba77ef058ac9c898243f6097"} Jun 02 01:41:51.923933 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:51.923902 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gt4w8" Jun 02 01:41:51.923933 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:51.923922 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-smqwh" Jun 02 01:41:51.924113 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:51.924002 2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gt4w8" podUID="7db70fbd-3197-47ed-a00b-d6b4fbb4c8a8" Jun 02 01:41:51.924187 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:51.924110 2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-smqwh" podUID="f3542744-21d2-4162-a2f9-4a474d18dc19" Jun 02 01:41:52.121963 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:52.121937 2571 generic.go:358] "Generic (PLEG): container finished" podID="ae3e1499-5744-4270-b94d-f292d30bd1b2" containerID="ab0da5006d6dcec2230c5e746452869cf8afbc0490fa19a33623905f076dfc53" exitCode=0 Jun 02 01:41:52.122263 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:52.122016 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tglfq" event={"ID":"ae3e1499-5744-4270-b94d-f292d30bd1b2","Type":"ContainerDied","Data":"ab0da5006d6dcec2230c5e746452869cf8afbc0490fa19a33623905f076dfc53"} Jun 02 01:41:53.923580 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:53.923543 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-smqwh" Jun 02 01:41:53.924034 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:53.923532 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gt4w8" Jun 02 01:41:53.924034 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:53.923690 2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-smqwh" podUID="f3542744-21d2-4162-a2f9-4a474d18dc19" Jun 02 01:41:53.924034 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:53.923752 2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gt4w8" podUID="7db70fbd-3197-47ed-a00b-d6b4fbb4c8a8" Jun 02 01:41:55.550897 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:55.550856 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3542744-21d2-4162-a2f9-4a474d18dc19-metrics-certs\") pod \"network-metrics-daemon-smqwh\" (UID: \"f3542744-21d2-4162-a2f9-4a474d18dc19\") " pod="openshift-multus/network-metrics-daemon-smqwh" Jun 02 01:41:55.551394 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:55.550984 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jun 02 01:41:55.551394 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:55.551058 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3542744-21d2-4162-a2f9-4a474d18dc19-metrics-certs podName:f3542744-21d2-4162-a2f9-4a474d18dc19 nodeName:}" failed. No retries permitted until 2026-06-02 01:42:27.551042229 +0000 UTC m=+65.196848849 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f3542744-21d2-4162-a2f9-4a474d18dc19-metrics-certs") pod "network-metrics-daemon-smqwh" (UID: "f3542744-21d2-4162-a2f9-4a474d18dc19") : object "openshift-multus"/"metrics-daemon-secret" not registered Jun 02 01:41:55.652070 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:55.652029 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6f7hb\" (UniqueName: \"kubernetes.io/projected/7db70fbd-3197-47ed-a00b-d6b4fbb4c8a8-kube-api-access-6f7hb\") pod \"network-check-target-gt4w8\" (UID: \"7db70fbd-3197-47ed-a00b-d6b4fbb4c8a8\") " pod="openshift-network-diagnostics/network-check-target-gt4w8" Jun 02 01:41:55.652246 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:55.652223 2571 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jun 02 01:41:55.652293 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:55.652248 2571 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jun 02 01:41:55.652293 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:55.652260 2571 projected.go:194] Error preparing data for projected volume kube-api-access-6f7hb for pod openshift-network-diagnostics/network-check-target-gt4w8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 02 01:41:55.652392 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:55.652324 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7db70fbd-3197-47ed-a00b-d6b4fbb4c8a8-kube-api-access-6f7hb podName:7db70fbd-3197-47ed-a00b-d6b4fbb4c8a8 nodeName:}" failed. No retries permitted until 2026-06-02 01:42:27.652306973 +0000 UTC m=+65.298113595 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-6f7hb" (UniqueName: "kubernetes.io/projected/7db70fbd-3197-47ed-a00b-d6b4fbb4c8a8-kube-api-access-6f7hb") pod "network-check-target-gt4w8" (UID: "7db70fbd-3197-47ed-a00b-d6b4fbb4c8a8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 02 01:41:55.924209 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:55.924175 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gt4w8" Jun 02 01:41:55.924404 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:55.924178 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-smqwh" Jun 02 01:41:55.924404 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:55.924287 2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gt4w8" podUID="7db70fbd-3197-47ed-a00b-d6b4fbb4c8a8" Jun 02 01:41:55.924404 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:55.924366 2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-smqwh" podUID="f3542744-21d2-4162-a2f9-4a474d18dc19" Jun 02 01:41:56.183722 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:56.183645 2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-117.ec2.internal" event="NodeReady" Jun 02 01:41:56.183880 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:56.183805 2571 kubelet_node_status.go:550] "Fast updating node status as it just became ready" Jun 02 01:41:56.229848 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:56.229817 2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-fs4kq"] Jun 02 01:41:56.237284 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:56.237250 2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-m4mvd"] Jun 02 01:41:56.237437 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:56.237420 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fs4kq" Jun 02 01:41:56.239822 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:56.239791 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jun 02 01:41:56.239822 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:56.239806 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-q2kdm\"" Jun 02 01:41:56.240184 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:56.239972 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jun 02 01:41:56.240755 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:56.240736 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-m4mvd" Jun 02 01:41:56.241404 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:56.241376 2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fs4kq"] Jun 02 01:41:56.242974 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:56.242932 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jun 02 01:41:56.243076 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:56.243004 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jun 02 01:41:56.243214 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:56.243192 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jun 02 01:41:56.243336 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:56.243198 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-4kzbv\"" Jun 02 01:41:56.245858 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:56.245824 2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-m4mvd"] Jun 02 01:41:56.355751 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:56.355716 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97195e1b-54fc-432f-a909-88dac67a6235-tmp-dir\") pod \"dns-default-fs4kq\" (UID: \"97195e1b-54fc-432f-a909-88dac67a6235\") " pod="openshift-dns/dns-default-fs4kq" Jun 02 01:41:56.355942 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:56.355811 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qftq\" (UniqueName: \"kubernetes.io/projected/97195e1b-54fc-432f-a909-88dac67a6235-kube-api-access-6qftq\") pod \"dns-default-fs4kq\" (UID: \"97195e1b-54fc-432f-a909-88dac67a6235\") " pod="openshift-dns/dns-default-fs4kq" Jun 02 01:41:56.355942 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:56.355854 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/97195e1b-54fc-432f-a909-88dac67a6235-config-volume\") pod \"dns-default-fs4kq\" (UID: \"97195e1b-54fc-432f-a909-88dac67a6235\") " pod="openshift-dns/dns-default-fs4kq" Jun 02 01:41:56.355942 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:56.355877 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/97195e1b-54fc-432f-a909-88dac67a6235-metrics-tls\") pod \"dns-default-fs4kq\" (UID: \"97195e1b-54fc-432f-a909-88dac67a6235\") " pod="openshift-dns/dns-default-fs4kq" Jun 02 01:41:56.356048 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:56.355945 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e5cbb2d6-6afa-4be3-9aab-2f3bd30125bd-cert\") pod \"ingress-canary-m4mvd\" (UID: \"e5cbb2d6-6afa-4be3-9aab-2f3bd30125bd\") " pod="openshift-ingress-canary/ingress-canary-m4mvd" Jun 02 01:41:56.356048 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:56.355974 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jthjk\" (UniqueName: \"kubernetes.io/projected/e5cbb2d6-6afa-4be3-9aab-2f3bd30125bd-kube-api-access-jthjk\") pod \"ingress-canary-m4mvd\" (UID: \"e5cbb2d6-6afa-4be3-9aab-2f3bd30125bd\") " pod="openshift-ingress-canary/ingress-canary-m4mvd" Jun 02 01:41:56.439301 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:56.439223 2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d55995968-5zw74"] Jun 02 01:41:56.456410 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:56.456382 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6qftq\" (UniqueName: \"kubernetes.io/projected/97195e1b-54fc-432f-a909-88dac67a6235-kube-api-access-6qftq\") pod \"dns-default-fs4kq\" (UID: \"97195e1b-54fc-432f-a909-88dac67a6235\") " pod="openshift-dns/dns-default-fs4kq" Jun 02 01:41:56.456600 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:56.456436 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/97195e1b-54fc-432f-a909-88dac67a6235-config-volume\") pod \"dns-default-fs4kq\" (UID: \"97195e1b-54fc-432f-a909-88dac67a6235\") " pod="openshift-dns/dns-default-fs4kq" Jun 02 01:41:56.456600 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:56.456479 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/97195e1b-54fc-432f-a909-88dac67a6235-metrics-tls\") pod \"dns-default-fs4kq\" (UID: \"97195e1b-54fc-432f-a909-88dac67a6235\") " pod="openshift-dns/dns-default-fs4kq" Jun 02 01:41:56.456600 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:56.456510 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e5cbb2d6-6afa-4be3-9aab-2f3bd30125bd-cert\") pod \"ingress-canary-m4mvd\" (UID: \"e5cbb2d6-6afa-4be3-9aab-2f3bd30125bd\") " pod="openshift-ingress-canary/ingress-canary-m4mvd" Jun 02 01:41:56.456600 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:56.456536 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jthjk\" (UniqueName: \"kubernetes.io/projected/e5cbb2d6-6afa-4be3-9aab-2f3bd30125bd-kube-api-access-jthjk\") pod \"ingress-canary-m4mvd\" (UID: \"e5cbb2d6-6afa-4be3-9aab-2f3bd30125bd\") " pod="openshift-ingress-canary/ingress-canary-m4mvd" Jun 02 01:41:56.456600 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:56.456567 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97195e1b-54fc-432f-a909-88dac67a6235-tmp-dir\") pod \"dns-default-fs4kq\" (UID: \"97195e1b-54fc-432f-a909-88dac67a6235\") " pod="openshift-dns/dns-default-fs4kq" Jun 02 01:41:56.456843 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:56.456632 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jun 02 01:41:56.456843 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:56.456701 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97195e1b-54fc-432f-a909-88dac67a6235-metrics-tls podName:97195e1b-54fc-432f-a909-88dac67a6235 nodeName:}" failed. No retries permitted until 2026-06-02 01:41:56.956680428 +0000 UTC m=+34.602487050 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/97195e1b-54fc-432f-a909-88dac67a6235-metrics-tls") pod "dns-default-fs4kq" (UID: "97195e1b-54fc-432f-a909-88dac67a6235") : secret "dns-default-metrics-tls" not found Jun 02 01:41:56.456843 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:56.456825 2571 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found Jun 02 01:41:56.456991 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:56.456839 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97195e1b-54fc-432f-a909-88dac67a6235-tmp-dir\") pod \"dns-default-fs4kq\" (UID: \"97195e1b-54fc-432f-a909-88dac67a6235\") " pod="openshift-dns/dns-default-fs4kq" Jun 02 01:41:56.456991 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:56.456884 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5cbb2d6-6afa-4be3-9aab-2f3bd30125bd-cert podName:e5cbb2d6-6afa-4be3-9aab-2f3bd30125bd nodeName:}" failed. No retries permitted until 2026-06-02 01:41:56.956866579 +0000 UTC m=+34.602673208 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e5cbb2d6-6afa-4be3-9aab-2f3bd30125bd-cert") pod "ingress-canary-m4mvd" (UID: "e5cbb2d6-6afa-4be3-9aab-2f3bd30125bd") : secret "canary-serving-cert" not found Jun 02 01:41:56.457311 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:56.457279 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/97195e1b-54fc-432f-a909-88dac67a6235-config-volume\") pod \"dns-default-fs4kq\" (UID: \"97195e1b-54fc-432f-a909-88dac67a6235\") " pod="openshift-dns/dns-default-fs4kq" Jun 02 01:41:56.462457 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:56.462423 2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d55995968-5zw74"] Jun 02 01:41:56.462591 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:56.462561 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d55995968-5zw74" Jun 02 01:41:56.464810 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:56.464790 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jun 02 01:41:56.465104 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:56.465082 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jun 02 01:41:56.465241 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:56.465215 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jun 02 01:41:56.465461 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:56.465344 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jun 02 01:41:56.465461 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:56.465378 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jun 02 01:41:56.465461 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:56.465380 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jun 02 01:41:56.465781 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:56.465547 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jun 02 01:41:56.480422 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:56.480398 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qftq\" (UniqueName: \"kubernetes.io/projected/97195e1b-54fc-432f-a909-88dac67a6235-kube-api-access-6qftq\") pod \"dns-default-fs4kq\" (UID: \"97195e1b-54fc-432f-a909-88dac67a6235\") " pod="openshift-dns/dns-default-fs4kq" Jun 02 01:41:56.480533 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:56.480483 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jthjk\" (UniqueName: \"kubernetes.io/projected/e5cbb2d6-6afa-4be3-9aab-2f3bd30125bd-kube-api-access-jthjk\") pod \"ingress-canary-m4mvd\" (UID: \"e5cbb2d6-6afa-4be3-9aab-2f3bd30125bd\") " pod="openshift-ingress-canary/ingress-canary-m4mvd" Jun 02 01:41:56.557692 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:56.557660 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ecb04a0f-96b4-46f0-baa3-cac95651f7bf-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5d55995968-5zw74\" (UID: \"ecb04a0f-96b4-46f0-baa3-cac95651f7bf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d55995968-5zw74" Jun 02 01:41:56.558153 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:56.557705 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ecb04a0f-96b4-46f0-baa3-cac95651f7bf-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5d55995968-5zw74\" (UID: \"ecb04a0f-96b4-46f0-baa3-cac95651f7bf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d55995968-5zw74" Jun 02 01:41:56.558153 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:56.557815 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ecb04a0f-96b4-46f0-baa3-cac95651f7bf-hub\") pod \"cluster-proxy-proxy-agent-5d55995968-5zw74\" (UID: \"ecb04a0f-96b4-46f0-baa3-cac95651f7bf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d55995968-5zw74" Jun 02 01:41:56.558153 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:56.557861 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkpt4\" (UniqueName: \"kubernetes.io/projected/ecb04a0f-96b4-46f0-baa3-cac95651f7bf-kube-api-access-bkpt4\") pod \"cluster-proxy-proxy-agent-5d55995968-5zw74\" (UID: \"ecb04a0f-96b4-46f0-baa3-cac95651f7bf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d55995968-5zw74" Jun 02 01:41:56.558153 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:56.557936 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ecb04a0f-96b4-46f0-baa3-cac95651f7bf-ca\") pod \"cluster-proxy-proxy-agent-5d55995968-5zw74\" (UID: \"ecb04a0f-96b4-46f0-baa3-cac95651f7bf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d55995968-5zw74" Jun 02 01:41:56.558153 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:56.557980 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/ecb04a0f-96b4-46f0-baa3-cac95651f7bf-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-5d55995968-5zw74\" (UID: \"ecb04a0f-96b4-46f0-baa3-cac95651f7bf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d55995968-5zw74" Jun 02 01:41:56.658602 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:56.658567 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ecb04a0f-96b4-46f0-baa3-cac95651f7bf-hub\") pod \"cluster-proxy-proxy-agent-5d55995968-5zw74\" (UID: \"ecb04a0f-96b4-46f0-baa3-cac95651f7bf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d55995968-5zw74" Jun 02 01:41:56.658602 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:56.658611 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bkpt4\" (UniqueName: \"kubernetes.io/projected/ecb04a0f-96b4-46f0-baa3-cac95651f7bf-kube-api-access-bkpt4\") pod \"cluster-proxy-proxy-agent-5d55995968-5zw74\" (UID: \"ecb04a0f-96b4-46f0-baa3-cac95651f7bf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d55995968-5zw74" Jun 02 01:41:56.658866 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:56.658652 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ecb04a0f-96b4-46f0-baa3-cac95651f7bf-ca\") pod \"cluster-proxy-proxy-agent-5d55995968-5zw74\" (UID: \"ecb04a0f-96b4-46f0-baa3-cac95651f7bf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d55995968-5zw74" Jun 02 01:41:56.658866 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:56.658681 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/ecb04a0f-96b4-46f0-baa3-cac95651f7bf-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-5d55995968-5zw74\" (UID: \"ecb04a0f-96b4-46f0-baa3-cac95651f7bf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d55995968-5zw74" Jun 02 01:41:56.658866 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:56.658712 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ecb04a0f-96b4-46f0-baa3-cac95651f7bf-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5d55995968-5zw74\" (UID: \"ecb04a0f-96b4-46f0-baa3-cac95651f7bf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d55995968-5zw74" Jun 02 01:41:56.658866 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:56.658748 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ecb04a0f-96b4-46f0-baa3-cac95651f7bf-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5d55995968-5zw74\" (UID: \"ecb04a0f-96b4-46f0-baa3-cac95651f7bf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d55995968-5zw74" Jun 02 01:41:56.659617 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:56.659569 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/ecb04a0f-96b4-46f0-baa3-cac95651f7bf-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-5d55995968-5zw74\" (UID: \"ecb04a0f-96b4-46f0-baa3-cac95651f7bf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d55995968-5zw74" Jun 02 01:41:56.661695 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:56.661670 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ecb04a0f-96b4-46f0-baa3-cac95651f7bf-ca\") pod \"cluster-proxy-proxy-agent-5d55995968-5zw74\" (UID: \"ecb04a0f-96b4-46f0-baa3-cac95651f7bf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d55995968-5zw74" Jun 02 01:41:56.661803 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:56.661725 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ecb04a0f-96b4-46f0-baa3-cac95651f7bf-hub\") pod \"cluster-proxy-proxy-agent-5d55995968-5zw74\" (UID: \"ecb04a0f-96b4-46f0-baa3-cac95651f7bf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d55995968-5zw74" Jun 02 01:41:56.661803 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:56.661747 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ecb04a0f-96b4-46f0-baa3-cac95651f7bf-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5d55995968-5zw74\" (UID: \"ecb04a0f-96b4-46f0-baa3-cac95651f7bf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d55995968-5zw74" Jun 02 01:41:56.661969 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:56.661922 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ecb04a0f-96b4-46f0-baa3-cac95651f7bf-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5d55995968-5zw74\" (UID: \"ecb04a0f-96b4-46f0-baa3-cac95651f7bf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d55995968-5zw74" Jun 02 01:41:56.666562 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:56.666531 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkpt4\" (UniqueName: \"kubernetes.io/projected/ecb04a0f-96b4-46f0-baa3-cac95651f7bf-kube-api-access-bkpt4\") pod \"cluster-proxy-proxy-agent-5d55995968-5zw74\" (UID: \"ecb04a0f-96b4-46f0-baa3-cac95651f7bf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d55995968-5zw74" Jun 02 01:41:56.783746 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:56.783487 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d55995968-5zw74" Jun 02 01:41:56.948466 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:56.948435 2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d55995968-5zw74"] Jun 02 01:41:56.953239 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:41:56.953211 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podecb04a0f_96b4_46f0_baa3_cac95651f7bf.slice/crio-a3c20a33e8708a7c8d0c1d9ef32833a39ffbd76e4d2c3478a64880e175664a92 WatchSource:0}: Error finding container a3c20a33e8708a7c8d0c1d9ef32833a39ffbd76e4d2c3478a64880e175664a92: Status 404 returned error can't find the container with id a3c20a33e8708a7c8d0c1d9ef32833a39ffbd76e4d2c3478a64880e175664a92 Jun 02 01:41:56.961119 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:56.961090 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/97195e1b-54fc-432f-a909-88dac67a6235-metrics-tls\") pod \"dns-default-fs4kq\" (UID: \"97195e1b-54fc-432f-a909-88dac67a6235\") " pod="openshift-dns/dns-default-fs4kq" Jun 02 01:41:56.961236 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:56.961156 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e5cbb2d6-6afa-4be3-9aab-2f3bd30125bd-cert\") pod \"ingress-canary-m4mvd\" (UID: \"e5cbb2d6-6afa-4be3-9aab-2f3bd30125bd\") " pod="openshift-ingress-canary/ingress-canary-m4mvd" Jun 02 01:41:56.961485 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:56.961243 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jun 02 01:41:56.961485 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:56.961283 2571 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found Jun 02 01:41:56.961485 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:56.961314 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97195e1b-54fc-432f-a909-88dac67a6235-metrics-tls podName:97195e1b-54fc-432f-a909-88dac67a6235 nodeName:}" failed. No retries permitted until 2026-06-02 01:41:57.961295739 +0000 UTC m=+35.607102369 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/97195e1b-54fc-432f-a909-88dac67a6235-metrics-tls") pod "dns-default-fs4kq" (UID: "97195e1b-54fc-432f-a909-88dac67a6235") : secret "dns-default-metrics-tls" not found Jun 02 01:41:56.961485 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:56.961330 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5cbb2d6-6afa-4be3-9aab-2f3bd30125bd-cert podName:e5cbb2d6-6afa-4be3-9aab-2f3bd30125bd nodeName:}" failed. No retries permitted until 2026-06-02 01:41:57.961323324 +0000 UTC m=+35.607129943 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e5cbb2d6-6afa-4be3-9aab-2f3bd30125bd-cert") pod "ingress-canary-m4mvd" (UID: "e5cbb2d6-6afa-4be3-9aab-2f3bd30125bd") : secret "canary-serving-cert" not found Jun 02 01:41:57.132402 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:57.132303 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d55995968-5zw74" event={"ID":"ecb04a0f-96b4-46f0-baa3-cac95651f7bf","Type":"ContainerStarted","Data":"a3c20a33e8708a7c8d0c1d9ef32833a39ffbd76e4d2c3478a64880e175664a92"} Jun 02 01:41:57.924125 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:57.924089 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gt4w8" Jun 02 01:41:57.924731 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:57.924094 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-smqwh" Jun 02 01:41:57.928222 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:57.928198 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6hbws\"" Jun 02 01:41:57.928355 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:57.928198 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jun 02 01:41:57.928355 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:57.928239 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xnph6\"" Jun 02 01:41:57.928355 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:57.928254 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jun 02 01:41:57.928355 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:57.928241 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jun 02 01:41:57.968100 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:57.968059 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/97195e1b-54fc-432f-a909-88dac67a6235-metrics-tls\") pod \"dns-default-fs4kq\" (UID: \"97195e1b-54fc-432f-a909-88dac67a6235\") " pod="openshift-dns/dns-default-fs4kq" Jun 02 01:41:57.968284 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:57.968110 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e5cbb2d6-6afa-4be3-9aab-2f3bd30125bd-cert\") pod \"ingress-canary-m4mvd\" (UID: \"e5cbb2d6-6afa-4be3-9aab-2f3bd30125bd\") " pod="openshift-ingress-canary/ingress-canary-m4mvd" Jun 02 01:41:57.968284 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:57.968240 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jun 02 01:41:57.968391 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:57.968315 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97195e1b-54fc-432f-a909-88dac67a6235-metrics-tls podName:97195e1b-54fc-432f-a909-88dac67a6235 nodeName:}" failed. No retries permitted until 2026-06-02 01:41:59.968295625 +0000 UTC m=+37.614102264 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/97195e1b-54fc-432f-a909-88dac67a6235-metrics-tls") pod "dns-default-fs4kq" (UID: "97195e1b-54fc-432f-a909-88dac67a6235") : secret "dns-default-metrics-tls" not found Jun 02 01:41:57.968391 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:57.968239 2571 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found Jun 02 01:41:57.968391 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:57.968390 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5cbb2d6-6afa-4be3-9aab-2f3bd30125bd-cert podName:e5cbb2d6-6afa-4be3-9aab-2f3bd30125bd nodeName:}" failed. No retries permitted until 2026-06-02 01:41:59.968377237 +0000 UTC m=+37.614183868 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e5cbb2d6-6afa-4be3-9aab-2f3bd30125bd-cert") pod "ingress-canary-m4mvd" (UID: "e5cbb2d6-6afa-4be3-9aab-2f3bd30125bd") : secret "canary-serving-cert" not found Jun 02 01:41:59.984983 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:59.984940 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/97195e1b-54fc-432f-a909-88dac67a6235-metrics-tls\") pod \"dns-default-fs4kq\" (UID: \"97195e1b-54fc-432f-a909-88dac67a6235\") " pod="openshift-dns/dns-default-fs4kq" Jun 02 01:41:59.985692 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:41:59.985002 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e5cbb2d6-6afa-4be3-9aab-2f3bd30125bd-cert\") pod \"ingress-canary-m4mvd\" (UID: \"e5cbb2d6-6afa-4be3-9aab-2f3bd30125bd\") " pod="openshift-ingress-canary/ingress-canary-m4mvd" Jun 02 01:41:59.985692 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:59.985150 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jun 02 01:41:59.985692 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:59.985189 2571 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found Jun 02 01:41:59.985692 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:59.985257 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97195e1b-54fc-432f-a909-88dac67a6235-metrics-tls podName:97195e1b-54fc-432f-a909-88dac67a6235 nodeName:}" failed. No retries permitted until 2026-06-02 01:42:03.985235797 +0000 UTC m=+41.631042430 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/97195e1b-54fc-432f-a909-88dac67a6235-metrics-tls") pod "dns-default-fs4kq" (UID: "97195e1b-54fc-432f-a909-88dac67a6235") : secret "dns-default-metrics-tls" not found Jun 02 01:41:59.985692 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:41:59.985279 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5cbb2d6-6afa-4be3-9aab-2f3bd30125bd-cert podName:e5cbb2d6-6afa-4be3-9aab-2f3bd30125bd nodeName:}" failed. No retries permitted until 2026-06-02 01:42:03.985269062 +0000 UTC m=+41.631075715 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e5cbb2d6-6afa-4be3-9aab-2f3bd30125bd-cert") pod "ingress-canary-m4mvd" (UID: "e5cbb2d6-6afa-4be3-9aab-2f3bd30125bd") : secret "canary-serving-cert" not found Jun 02 01:42:00.142093 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:42:00.142061 2571 generic.go:358] "Generic (PLEG): container finished" podID="ae3e1499-5744-4270-b94d-f292d30bd1b2" containerID="e1b985aebaf97f9b1c2433c764be628f7882ca64e81e15800c0df7dee246e067" exitCode=0 Jun 02 01:42:00.142237 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:42:00.142120 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tglfq" event={"ID":"ae3e1499-5744-4270-b94d-f292d30bd1b2","Type":"ContainerDied","Data":"e1b985aebaf97f9b1c2433c764be628f7882ca64e81e15800c0df7dee246e067"} Jun 02 01:42:01.148421 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:42:01.148363 2571 generic.go:358] "Generic (PLEG): container finished" podID="ae3e1499-5744-4270-b94d-f292d30bd1b2" containerID="9fb57632660d07728bc0fc0810cf162d37bb20ec87f30ba71e1081b4206cdf8b" exitCode=0 Jun 02 01:42:01.148421 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:42:01.148411 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tglfq" event={"ID":"ae3e1499-5744-4270-b94d-f292d30bd1b2","Type":"ContainerDied","Data":"9fb57632660d07728bc0fc0810cf162d37bb20ec87f30ba71e1081b4206cdf8b"} Jun 02 01:42:02.153565 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:42:02.153531 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tglfq" event={"ID":"ae3e1499-5744-4270-b94d-f292d30bd1b2","Type":"ContainerStarted","Data":"55272ecddf882feb35874fc38c8b78e3b51d862d19676c7c419b2b1ba2273368"} Jun 02 01:42:03.157914 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:42:03.157878 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d55995968-5zw74" event={"ID":"ecb04a0f-96b4-46f0-baa3-cac95651f7bf","Type":"ContainerStarted","Data":"9d16f1b54cc2ec17332f60205ddcda8d1564ea5db5897ae3fe406feff84b2148"} Jun 02 01:42:03.179598 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:42:03.179474 2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-tglfq" podStartSLOduration=4.545650831 podStartE2EDuration="40.179455704s" podCreationTimestamp="2026-06-02 01:41:23 +0000 UTC" firstStartedPulling="2026-06-02 01:41:24.171404426 +0000 UTC m=+1.817211046" lastFinishedPulling="2026-06-02 01:41:59.805209288 +0000 UTC m=+37.451015919" observedRunningTime="2026-06-02 01:42:03.178305679 +0000 UTC m=+40.824112322" watchObservedRunningTime="2026-06-02 01:42:03.179455704 +0000 UTC m=+40.825262345" Jun 02 01:42:04.017147 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:42:04.017082 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/97195e1b-54fc-432f-a909-88dac67a6235-metrics-tls\") pod \"dns-default-fs4kq\" (UID: \"97195e1b-54fc-432f-a909-88dac67a6235\") " pod="openshift-dns/dns-default-fs4kq" Jun 02 01:42:04.017338 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:42:04.017170 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e5cbb2d6-6afa-4be3-9aab-2f3bd30125bd-cert\") pod \"ingress-canary-m4mvd\" (UID: \"e5cbb2d6-6afa-4be3-9aab-2f3bd30125bd\") " pod="openshift-ingress-canary/ingress-canary-m4mvd" Jun 02 01:42:04.017338 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:42:04.017267 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jun 02 01:42:04.017533 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:42:04.017345 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97195e1b-54fc-432f-a909-88dac67a6235-metrics-tls podName:97195e1b-54fc-432f-a909-88dac67a6235 nodeName:}" failed. No retries permitted until 2026-06-02 01:42:12.017327358 +0000 UTC m=+49.663133977 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/97195e1b-54fc-432f-a909-88dac67a6235-metrics-tls") pod "dns-default-fs4kq" (UID: "97195e1b-54fc-432f-a909-88dac67a6235") : secret "dns-default-metrics-tls" not found Jun 02 01:42:04.017533 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:42:04.017357 2571 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found Jun 02 01:42:04.017533 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:42:04.017415 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5cbb2d6-6afa-4be3-9aab-2f3bd30125bd-cert podName:e5cbb2d6-6afa-4be3-9aab-2f3bd30125bd nodeName:}" failed. No retries permitted until 2026-06-02 01:42:12.017398713 +0000 UTC m=+49.663205346 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e5cbb2d6-6afa-4be3-9aab-2f3bd30125bd-cert") pod "ingress-canary-m4mvd" (UID: "e5cbb2d6-6afa-4be3-9aab-2f3bd30125bd") : secret "canary-serving-cert" not found Jun 02 01:42:06.165006 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:42:06.164970 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d55995968-5zw74" event={"ID":"ecb04a0f-96b4-46f0-baa3-cac95651f7bf","Type":"ContainerStarted","Data":"e6dd72a3f5c07e4b39333baab0f4db8f949b9d606c5a63aefd5d9c95a5b2b07b"} Jun 02 01:42:06.165392 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:42:06.165014 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d55995968-5zw74" event={"ID":"ecb04a0f-96b4-46f0-baa3-cac95651f7bf","Type":"ContainerStarted","Data":"ab4f7d7bc1045bbd740df4f8ec8abcd03cc173c6b8f6ac91d1f6061272ec453c"} Jun 02 01:42:06.182754 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:42:06.182708 2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d55995968-5zw74" podStartSLOduration=1.276953182 podStartE2EDuration="10.182694687s" podCreationTimestamp="2026-06-02 01:41:56 +0000 UTC" firstStartedPulling="2026-06-02 01:41:56.955161342 +0000 UTC m=+34.600967968" lastFinishedPulling="2026-06-02 01:42:05.86090285 +0000 UTC m=+43.506709473" observedRunningTime="2026-06-02 01:42:06.181153384 +0000 UTC m=+43.826960020" watchObservedRunningTime="2026-06-02 01:42:06.182694687 +0000 UTC m=+43.828501342" Jun 02 01:42:12.075213 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:42:12.075165 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/97195e1b-54fc-432f-a909-88dac67a6235-metrics-tls\") pod \"dns-default-fs4kq\" (UID: \"97195e1b-54fc-432f-a909-88dac67a6235\") " pod="openshift-dns/dns-default-fs4kq" Jun 02 01:42:12.075213 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:42:12.075216 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e5cbb2d6-6afa-4be3-9aab-2f3bd30125bd-cert\") pod \"ingress-canary-m4mvd\" (UID: \"e5cbb2d6-6afa-4be3-9aab-2f3bd30125bd\") " pod="openshift-ingress-canary/ingress-canary-m4mvd" Jun 02 01:42:12.075789 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:42:12.075328 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jun 02 01:42:12.075789 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:42:12.075343 2571 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found Jun 02 01:42:12.075789 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:42:12.075398 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97195e1b-54fc-432f-a909-88dac67a6235-metrics-tls podName:97195e1b-54fc-432f-a909-88dac67a6235 nodeName:}" failed. No retries permitted until 2026-06-02 01:42:28.075381589 +0000 UTC m=+65.721188210 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/97195e1b-54fc-432f-a909-88dac67a6235-metrics-tls") pod "dns-default-fs4kq" (UID: "97195e1b-54fc-432f-a909-88dac67a6235") : secret "dns-default-metrics-tls" not found Jun 02 01:42:12.075789 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:42:12.075414 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5cbb2d6-6afa-4be3-9aab-2f3bd30125bd-cert podName:e5cbb2d6-6afa-4be3-9aab-2f3bd30125bd nodeName:}" failed. No retries permitted until 2026-06-02 01:42:28.075406792 +0000 UTC m=+65.721213411 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e5cbb2d6-6afa-4be3-9aab-2f3bd30125bd-cert") pod "ingress-canary-m4mvd" (UID: "e5cbb2d6-6afa-4be3-9aab-2f3bd30125bd") : secret "canary-serving-cert" not found Jun 02 01:42:22.132125 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:42:22.132090 2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fvdb7" Jun 02 01:42:27.581628 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:42:27.581586 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3542744-21d2-4162-a2f9-4a474d18dc19-metrics-certs\") pod \"network-metrics-daemon-smqwh\" (UID: \"f3542744-21d2-4162-a2f9-4a474d18dc19\") " pod="openshift-multus/network-metrics-daemon-smqwh" Jun 02 01:42:27.584518 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:42:27.584497 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jun 02 01:42:27.592631 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:42:27.592606 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jun 02 01:42:27.592688 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:42:27.592677 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3542744-21d2-4162-a2f9-4a474d18dc19-metrics-certs podName:f3542744-21d2-4162-a2f9-4a474d18dc19 nodeName:}" failed. No retries permitted until 2026-06-02 01:43:31.592661717 +0000 UTC m=+129.238468337 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f3542744-21d2-4162-a2f9-4a474d18dc19-metrics-certs") pod "network-metrics-daemon-smqwh" (UID: "f3542744-21d2-4162-a2f9-4a474d18dc19") : secret "metrics-daemon-secret" not found Jun 02 01:42:27.682020 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:42:27.681989 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6f7hb\" (UniqueName: \"kubernetes.io/projected/7db70fbd-3197-47ed-a00b-d6b4fbb4c8a8-kube-api-access-6f7hb\") pod \"network-check-target-gt4w8\" (UID: \"7db70fbd-3197-47ed-a00b-d6b4fbb4c8a8\") " pod="openshift-network-diagnostics/network-check-target-gt4w8" Jun 02 01:42:27.684827 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:42:27.684809 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jun 02 01:42:27.695925 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:42:27.695908 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jun 02 01:42:27.706803 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:42:27.706767 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f7hb\" (UniqueName: \"kubernetes.io/projected/7db70fbd-3197-47ed-a00b-d6b4fbb4c8a8-kube-api-access-6f7hb\") pod \"network-check-target-gt4w8\" (UID: \"7db70fbd-3197-47ed-a00b-d6b4fbb4c8a8\") " pod="openshift-network-diagnostics/network-check-target-gt4w8" Jun 02 01:42:27.940255 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:42:27.940231 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xnph6\"" Jun 02 01:42:27.947033 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:42:27.947013 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gt4w8" Jun 02 01:42:28.065499 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:42:28.065469 2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gt4w8"] Jun 02 01:42:28.069082 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:42:28.069051 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7db70fbd_3197_47ed_a00b_d6b4fbb4c8a8.slice/crio-696c76df7bf1118420cb6001d7bdc6deaca77b654f02efadba9ebd3a01429293 WatchSource:0}: Error finding container 696c76df7bf1118420cb6001d7bdc6deaca77b654f02efadba9ebd3a01429293: Status 404 returned error can't find the container with id 696c76df7bf1118420cb6001d7bdc6deaca77b654f02efadba9ebd3a01429293 Jun 02 01:42:28.084361 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:42:28.084331 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/97195e1b-54fc-432f-a909-88dac67a6235-metrics-tls\") pod \"dns-default-fs4kq\" (UID: \"97195e1b-54fc-432f-a909-88dac67a6235\") " pod="openshift-dns/dns-default-fs4kq" Jun 02 01:42:28.084459 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:42:28.084373 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e5cbb2d6-6afa-4be3-9aab-2f3bd30125bd-cert\") pod \"ingress-canary-m4mvd\" (UID: \"e5cbb2d6-6afa-4be3-9aab-2f3bd30125bd\") " pod="openshift-ingress-canary/ingress-canary-m4mvd" Jun 02 01:42:28.084502 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:42:28.084488 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jun 02 01:42:28.084552 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:42:28.084542 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97195e1b-54fc-432f-a909-88dac67a6235-metrics-tls podName:97195e1b-54fc-432f-a909-88dac67a6235 nodeName:}" failed. No retries permitted until 2026-06-02 01:43:00.084526162 +0000 UTC m=+97.730332786 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/97195e1b-54fc-432f-a909-88dac67a6235-metrics-tls") pod "dns-default-fs4kq" (UID: "97195e1b-54fc-432f-a909-88dac67a6235") : secret "dns-default-metrics-tls" not found Jun 02 01:42:28.084603 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:42:28.084491 2571 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found Jun 02 01:42:28.084645 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:42:28.084622 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5cbb2d6-6afa-4be3-9aab-2f3bd30125bd-cert podName:e5cbb2d6-6afa-4be3-9aab-2f3bd30125bd nodeName:}" failed. No retries permitted until 2026-06-02 01:43:00.084610434 +0000 UTC m=+97.730417054 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e5cbb2d6-6afa-4be3-9aab-2f3bd30125bd-cert") pod "ingress-canary-m4mvd" (UID: "e5cbb2d6-6afa-4be3-9aab-2f3bd30125bd") : secret "canary-serving-cert" not found Jun 02 01:42:28.207440 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:42:28.207359 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gt4w8" event={"ID":"7db70fbd-3197-47ed-a00b-d6b4fbb4c8a8","Type":"ContainerStarted","Data":"696c76df7bf1118420cb6001d7bdc6deaca77b654f02efadba9ebd3a01429293"} Jun 02 01:42:32.216690 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:42:32.216642 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gt4w8" event={"ID":"7db70fbd-3197-47ed-a00b-d6b4fbb4c8a8","Type":"ContainerStarted","Data":"7206ffba02befbaef532c12d72186728d18e1dc6f0e55a3a9d64b9983f8ce3a2"} Jun 02 01:42:32.217078 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:42:32.216828 2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gt4w8" Jun 02 01:42:32.231712 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:42:32.231664 2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-gt4w8" podStartSLOduration=66.053907636 podStartE2EDuration="1m9.231653881s" podCreationTimestamp="2026-06-02 01:41:23 +0000 UTC" firstStartedPulling="2026-06-02 01:42:28.070926995 +0000 UTC m=+65.716733616" lastFinishedPulling="2026-06-02 01:42:31.248673238 +0000 UTC m=+68.894479861" observedRunningTime="2026-06-02 01:42:32.231246195 +0000 UTC m=+69.877052836" watchObservedRunningTime="2026-06-02 01:42:32.231653881 +0000 UTC m=+69.877460520" Jun 02 01:43:00.105020 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:00.104985 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/97195e1b-54fc-432f-a909-88dac67a6235-metrics-tls\") pod \"dns-default-fs4kq\" (UID: \"97195e1b-54fc-432f-a909-88dac67a6235\") " pod="openshift-dns/dns-default-fs4kq" Jun 02 01:43:00.105020 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:00.105024 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e5cbb2d6-6afa-4be3-9aab-2f3bd30125bd-cert\") pod \"ingress-canary-m4mvd\" (UID: \"e5cbb2d6-6afa-4be3-9aab-2f3bd30125bd\") " pod="openshift-ingress-canary/ingress-canary-m4mvd" Jun 02 01:43:00.105548 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:43:00.105145 2571 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found Jun 02 01:43:00.105548 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:43:00.105156 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jun 02 01:43:00.105548 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:43:00.105206 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5cbb2d6-6afa-4be3-9aab-2f3bd30125bd-cert podName:e5cbb2d6-6afa-4be3-9aab-2f3bd30125bd nodeName:}" failed. No retries permitted until 2026-06-02 01:44:04.10519285 +0000 UTC m=+161.750999470 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e5cbb2d6-6afa-4be3-9aab-2f3bd30125bd-cert") pod "ingress-canary-m4mvd" (UID: "e5cbb2d6-6afa-4be3-9aab-2f3bd30125bd") : secret "canary-serving-cert" not found Jun 02 01:43:00.105548 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:43:00.105219 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97195e1b-54fc-432f-a909-88dac67a6235-metrics-tls podName:97195e1b-54fc-432f-a909-88dac67a6235 nodeName:}" failed. No retries permitted until 2026-06-02 01:44:04.10521298 +0000 UTC m=+161.751019600 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/97195e1b-54fc-432f-a909-88dac67a6235-metrics-tls") pod "dns-default-fs4kq" (UID: "97195e1b-54fc-432f-a909-88dac67a6235") : secret "dns-default-metrics-tls" not found Jun 02 01:43:03.220929 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:03.220896 2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-gt4w8" Jun 02 01:43:13.781579 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:13.781532 2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-69877444c-rzbt6"] Jun 02 01:43:13.784227 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:13.784209 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-69877444c-rzbt6" Jun 02 01:43:13.787104 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:13.787086 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jun 02 01:43:13.787391 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:13.787372 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-hfb94\"" Jun 02 01:43:13.787778 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:13.787761 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jun 02 01:43:13.787840 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:13.787831 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jun 02 01:43:13.788318 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:13.788304 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jun 02 01:43:13.799830 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:13.799806 2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-69877444c-rzbt6"] Jun 02 01:43:13.899631 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:13.899596 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/48a4fcb7-07a8-4a1a-a4bb-ca854f1bd6c0-config\") pod \"service-ca-operator-69877444c-rzbt6\" (UID: \"48a4fcb7-07a8-4a1a-a4bb-ca854f1bd6c0\") " pod="openshift-service-ca-operator/service-ca-operator-69877444c-rzbt6" Jun 02 01:43:13.899631 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:13.899649 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj8kv\" (UniqueName: \"kubernetes.io/projected/48a4fcb7-07a8-4a1a-a4bb-ca854f1bd6c0-kube-api-access-nj8kv\") pod \"service-ca-operator-69877444c-rzbt6\" (UID: \"48a4fcb7-07a8-4a1a-a4bb-ca854f1bd6c0\") " pod="openshift-service-ca-operator/service-ca-operator-69877444c-rzbt6" Jun 02 01:43:13.899859 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:13.899733 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/48a4fcb7-07a8-4a1a-a4bb-ca854f1bd6c0-serving-cert\") pod \"service-ca-operator-69877444c-rzbt6\" (UID: \"48a4fcb7-07a8-4a1a-a4bb-ca854f1bd6c0\") " pod="openshift-service-ca-operator/service-ca-operator-69877444c-rzbt6" Jun 02 01:43:14.007462 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:14.007430 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/48a4fcb7-07a8-4a1a-a4bb-ca854f1bd6c0-config\") pod \"service-ca-operator-69877444c-rzbt6\" (UID: \"48a4fcb7-07a8-4a1a-a4bb-ca854f1bd6c0\") " pod="openshift-service-ca-operator/service-ca-operator-69877444c-rzbt6" Jun 02 01:43:14.007598 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:14.007473 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nj8kv\" (UniqueName: \"kubernetes.io/projected/48a4fcb7-07a8-4a1a-a4bb-ca854f1bd6c0-kube-api-access-nj8kv\") pod \"service-ca-operator-69877444c-rzbt6\" (UID: \"48a4fcb7-07a8-4a1a-a4bb-ca854f1bd6c0\") " pod="openshift-service-ca-operator/service-ca-operator-69877444c-rzbt6" Jun 02 01:43:14.007662 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:14.007600 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/48a4fcb7-07a8-4a1a-a4bb-ca854f1bd6c0-serving-cert\") pod \"service-ca-operator-69877444c-rzbt6\" (UID: \"48a4fcb7-07a8-4a1a-a4bb-ca854f1bd6c0\") " pod="openshift-service-ca-operator/service-ca-operator-69877444c-rzbt6" Jun 02 01:43:14.008075 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:14.008057 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/48a4fcb7-07a8-4a1a-a4bb-ca854f1bd6c0-config\") pod \"service-ca-operator-69877444c-rzbt6\" (UID: \"48a4fcb7-07a8-4a1a-a4bb-ca854f1bd6c0\") " pod="openshift-service-ca-operator/service-ca-operator-69877444c-rzbt6" Jun 02 01:43:14.010014 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:14.009990 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/48a4fcb7-07a8-4a1a-a4bb-ca854f1bd6c0-serving-cert\") pod \"service-ca-operator-69877444c-rzbt6\" (UID: \"48a4fcb7-07a8-4a1a-a4bb-ca854f1bd6c0\") " pod="openshift-service-ca-operator/service-ca-operator-69877444c-rzbt6" Jun 02 01:43:14.015663 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:14.015639 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj8kv\" (UniqueName: \"kubernetes.io/projected/48a4fcb7-07a8-4a1a-a4bb-ca854f1bd6c0-kube-api-access-nj8kv\") pod \"service-ca-operator-69877444c-rzbt6\" (UID: \"48a4fcb7-07a8-4a1a-a4bb-ca854f1bd6c0\") " pod="openshift-service-ca-operator/service-ca-operator-69877444c-rzbt6" Jun 02 01:43:14.093401 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:14.093294 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-69877444c-rzbt6" Jun 02 01:43:14.216353 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:14.216258 2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-69877444c-rzbt6"] Jun 02 01:43:14.218957 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:43:14.218927 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48a4fcb7_07a8_4a1a_a4bb_ca854f1bd6c0.slice/crio-2abfde9c75245da37df756eb9e885391ff53c2bcff339f12d20d62849a590c00 WatchSource:0}: Error finding container 2abfde9c75245da37df756eb9e885391ff53c2bcff339f12d20d62849a590c00: Status 404 returned error can't find the container with id 2abfde9c75245da37df756eb9e885391ff53c2bcff339f12d20d62849a590c00 Jun 02 01:43:14.294651 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:14.294620 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-69877444c-rzbt6" event={"ID":"48a4fcb7-07a8-4a1a-a4bb-ca854f1bd6c0","Type":"ContainerStarted","Data":"2abfde9c75245da37df756eb9e885391ff53c2bcff339f12d20d62849a590c00"} Jun 02 01:43:16.300004 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:16.299910 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-69877444c-rzbt6" event={"ID":"48a4fcb7-07a8-4a1a-a4bb-ca854f1bd6c0","Type":"ContainerStarted","Data":"8f836fc4b6c57901ddeb371c372c231ce4cf2e0544b1cce22a6d079e06a28909"} Jun 02 01:43:16.315465 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:16.315417 2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-69877444c-rzbt6" podStartSLOduration=1.554075826 podStartE2EDuration="3.315403473s" podCreationTimestamp="2026-06-02 01:43:13 +0000 UTC" firstStartedPulling="2026-06-02 01:43:14.220867032 +0000 UTC m=+111.866673652" lastFinishedPulling="2026-06-02 01:43:15.982194676 +0000 UTC m=+113.628001299" observedRunningTime="2026-06-02 01:43:16.314163841 +0000 UTC m=+113.959970490" watchObservedRunningTime="2026-06-02 01:43:16.315403473 +0000 UTC m=+113.961210115" Jun 02 01:43:19.331450 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:19.331423 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nm89v_5c45e264-330c-4438-bdc1-817a0a8b1b77/dns-node-resolver/0.log" Jun 02 01:43:20.331522 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:20.331493 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6rzvj_1fc82c63-2678-486d-9393-c3ea7f27abce/node-ca/0.log" Jun 02 01:43:20.463613 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:20.463577 2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-685df94d57-9xzqd"] Jun 02 01:43:20.465329 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:20.465314 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-685df94d57-9xzqd" Jun 02 01:43:20.467882 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:20.467859 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jun 02 01:43:20.467882 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:20.467871 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jun 02 01:43:20.468019 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:20.467881 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jun 02 01:43:20.468838 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:20.468822 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jun 02 01:43:20.468911 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:20.468822 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-2bdk8\"" Jun 02 01:43:20.475558 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:20.475537 2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-685df94d57-9xzqd"] Jun 02 01:43:20.553423 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:20.553384 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9d5f2925-510f-4c0e-922f-d92f6179ee2e-signing-key\") pod \"service-ca-685df94d57-9xzqd\" (UID: \"9d5f2925-510f-4c0e-922f-d92f6179ee2e\") " pod="openshift-service-ca/service-ca-685df94d57-9xzqd" Jun 02 01:43:20.553586 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:20.553466 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9d5f2925-510f-4c0e-922f-d92f6179ee2e-signing-cabundle\") pod \"service-ca-685df94d57-9xzqd\" (UID: \"9d5f2925-510f-4c0e-922f-d92f6179ee2e\") " pod="openshift-service-ca/service-ca-685df94d57-9xzqd" Jun 02 01:43:20.553586 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:20.553496 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6pcr\" (UniqueName: \"kubernetes.io/projected/9d5f2925-510f-4c0e-922f-d92f6179ee2e-kube-api-access-k6pcr\") pod \"service-ca-685df94d57-9xzqd\" (UID: \"9d5f2925-510f-4c0e-922f-d92f6179ee2e\") " pod="openshift-service-ca/service-ca-685df94d57-9xzqd" Jun 02 01:43:20.654233 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:20.654199 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-k6pcr\" (UniqueName: \"kubernetes.io/projected/9d5f2925-510f-4c0e-922f-d92f6179ee2e-kube-api-access-k6pcr\") pod \"service-ca-685df94d57-9xzqd\" (UID: \"9d5f2925-510f-4c0e-922f-d92f6179ee2e\") " pod="openshift-service-ca/service-ca-685df94d57-9xzqd" Jun 02 01:43:20.654434 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:20.654324 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9d5f2925-510f-4c0e-922f-d92f6179ee2e-signing-key\") pod \"service-ca-685df94d57-9xzqd\" (UID: \"9d5f2925-510f-4c0e-922f-d92f6179ee2e\") " pod="openshift-service-ca/service-ca-685df94d57-9xzqd" Jun 02 01:43:20.654434 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:20.654413 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9d5f2925-510f-4c0e-922f-d92f6179ee2e-signing-cabundle\") pod \"service-ca-685df94d57-9xzqd\" (UID: \"9d5f2925-510f-4c0e-922f-d92f6179ee2e\") " pod="openshift-service-ca/service-ca-685df94d57-9xzqd" Jun 02 01:43:20.657100 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:20.655855 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9d5f2925-510f-4c0e-922f-d92f6179ee2e-signing-cabundle\") pod \"service-ca-685df94d57-9xzqd\" (UID: \"9d5f2925-510f-4c0e-922f-d92f6179ee2e\") " pod="openshift-service-ca/service-ca-685df94d57-9xzqd" Jun 02 01:43:20.657636 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:20.657601 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9d5f2925-510f-4c0e-922f-d92f6179ee2e-signing-key\") pod \"service-ca-685df94d57-9xzqd\" (UID: \"9d5f2925-510f-4c0e-922f-d92f6179ee2e\") " pod="openshift-service-ca/service-ca-685df94d57-9xzqd" Jun 02 01:43:20.664143 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:20.664102 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6pcr\" (UniqueName: \"kubernetes.io/projected/9d5f2925-510f-4c0e-922f-d92f6179ee2e-kube-api-access-k6pcr\") pod \"service-ca-685df94d57-9xzqd\" (UID: \"9d5f2925-510f-4c0e-922f-d92f6179ee2e\") " pod="openshift-service-ca/service-ca-685df94d57-9xzqd" Jun 02 01:43:20.773674 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:20.773636 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-685df94d57-9xzqd" Jun 02 01:43:20.892259 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:20.892229 2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-685df94d57-9xzqd"] Jun 02 01:43:20.895252 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:43:20.895220 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d5f2925_510f_4c0e_922f_d92f6179ee2e.slice/crio-92d0e903122317e9502c4874adb1a9b9553d5e696a006918ce50018e1728ef50 WatchSource:0}: Error finding container 92d0e903122317e9502c4874adb1a9b9553d5e696a006918ce50018e1728ef50: Status 404 returned error can't find the container with id 92d0e903122317e9502c4874adb1a9b9553d5e696a006918ce50018e1728ef50 Jun 02 01:43:21.311489 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:21.311392 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-685df94d57-9xzqd" event={"ID":"9d5f2925-510f-4c0e-922f-d92f6179ee2e","Type":"ContainerStarted","Data":"505bc5a8d5f5e6ee3dedbdc59392a00f40a6f5d93915a00ade2b1ec1de49a135"} Jun 02 01:43:21.311489 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:21.311427 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-685df94d57-9xzqd" event={"ID":"9d5f2925-510f-4c0e-922f-d92f6179ee2e","Type":"ContainerStarted","Data":"92d0e903122317e9502c4874adb1a9b9553d5e696a006918ce50018e1728ef50"} Jun 02 01:43:21.327836 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:21.327795 2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-service-ca/service-ca-685df94d57-9xzqd" podStartSLOduration=1.327782596 podStartE2EDuration="1.327782596s" podCreationTimestamp="2026-06-02 01:43:20 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 01:43:21.327164396 +0000 UTC m=+118.972971035" watchObservedRunningTime="2026-06-02 01:43:21.327782596 +0000 UTC m=+118.973589239" Jun 02 01:43:31.640143 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:31.640085 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3542744-21d2-4162-a2f9-4a474d18dc19-metrics-certs\") pod \"network-metrics-daemon-smqwh\" (UID: \"f3542744-21d2-4162-a2f9-4a474d18dc19\") " pod="openshift-multus/network-metrics-daemon-smqwh" Jun 02 01:43:31.642437 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:31.642413 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3542744-21d2-4162-a2f9-4a474d18dc19-metrics-certs\") pod \"network-metrics-daemon-smqwh\" (UID: \"f3542744-21d2-4162-a2f9-4a474d18dc19\") " pod="openshift-multus/network-metrics-daemon-smqwh" Jun 02 01:43:31.846209 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:31.846178 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6hbws\"" Jun 02 01:43:31.854249 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:31.854231 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-smqwh" Jun 02 01:43:31.971209 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:31.971177 2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-smqwh"] Jun 02 01:43:31.974505 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:43:31.974475 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3542744_21d2_4162_a2f9_4a474d18dc19.slice/crio-e64017b531b13ee005109f03189ea9f9662d326f56a6cb544043c35828c479a6 WatchSource:0}: Error finding container e64017b531b13ee005109f03189ea9f9662d326f56a6cb544043c35828c479a6: Status 404 returned error can't find the container with id e64017b531b13ee005109f03189ea9f9662d326f56a6cb544043c35828c479a6 Jun 02 01:43:32.337197 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:32.337101 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-smqwh" event={"ID":"f3542744-21d2-4162-a2f9-4a474d18dc19","Type":"ContainerStarted","Data":"e64017b531b13ee005109f03189ea9f9662d326f56a6cb544043c35828c479a6"} Jun 02 01:43:33.341797 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:33.341709 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-smqwh" event={"ID":"f3542744-21d2-4162-a2f9-4a474d18dc19","Type":"ContainerStarted","Data":"d71b144f38e8425bfbcad343a48e69f36efe65b6c99d40e927ee50c964ffd598"} Jun 02 01:43:33.341797 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:33.341746 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-smqwh" event={"ID":"f3542744-21d2-4162-a2f9-4a474d18dc19","Type":"ContainerStarted","Data":"b44619d5f762aaf15a467844c20d32f80b04b8bd58749a442788c933a1a6571a"} Jun 02 01:43:33.356796 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:33.356756 2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-smqwh" podStartSLOduration=129.43901924 podStartE2EDuration="2m10.356742209s" podCreationTimestamp="2026-06-02 01:41:23 +0000 UTC" firstStartedPulling="2026-06-02 01:43:31.976378256 +0000 UTC m=+129.622184890" lastFinishedPulling="2026-06-02 01:43:32.894101224 +0000 UTC m=+130.539907859" observedRunningTime="2026-06-02 01:43:33.356021896 +0000 UTC m=+131.001828538" watchObservedRunningTime="2026-06-02 01:43:33.356742209 +0000 UTC m=+131.002548851" Jun 02 01:43:38.186047 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:38.186013 2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-vmqlc"] Jun 02 01:43:38.189152 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:38.189110 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vmqlc" Jun 02 01:43:38.192063 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:38.192042 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jun 02 01:43:38.192212 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:38.192063 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jun 02 01:43:38.193213 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:38.193192 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jun 02 01:43:38.193275 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:38.193218 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jun 02 01:43:38.193275 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:38.193227 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8qqcg\"" Jun 02 01:43:38.204597 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:38.204565 2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vmqlc"] Jun 02 01:43:38.228810 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:38.228776 2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-657f6fdbb8-cntl6"] Jun 02 01:43:38.230740 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:38.230720 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-657f6fdbb8-cntl6" Jun 02 01:43:38.233521 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:38.233500 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jun 02 01:43:38.233624 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:38.233549 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4h6ct\"" Jun 02 01:43:38.233672 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:38.233634 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jun 02 01:43:38.233777 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:38.233763 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jun 02 01:43:38.238524 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:38.238503 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jun 02 01:43:38.245984 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:38.245963 2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-657f6fdbb8-cntl6"] Jun 02 01:43:38.289597 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:38.289568 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/62166777-842d-49dc-9215-7b408dff79c9-data-volume\") pod \"insights-runtime-extractor-vmqlc\" (UID: \"62166777-842d-49dc-9215-7b408dff79c9\") " pod="openshift-insights/insights-runtime-extractor-vmqlc" Jun 02 01:43:38.289767 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:38.289631 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/62166777-842d-49dc-9215-7b408dff79c9-crio-socket\") pod \"insights-runtime-extractor-vmqlc\" (UID: \"62166777-842d-49dc-9215-7b408dff79c9\") " pod="openshift-insights/insights-runtime-extractor-vmqlc" Jun 02 01:43:38.289767 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:38.289658 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/62166777-842d-49dc-9215-7b408dff79c9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vmqlc\" (UID: \"62166777-842d-49dc-9215-7b408dff79c9\") " pod="openshift-insights/insights-runtime-extractor-vmqlc" Jun 02 01:43:38.289767 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:38.289676 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v245p\" (UniqueName: \"kubernetes.io/projected/62166777-842d-49dc-9215-7b408dff79c9-kube-api-access-v245p\") pod \"insights-runtime-extractor-vmqlc\" (UID: \"62166777-842d-49dc-9215-7b408dff79c9\") " pod="openshift-insights/insights-runtime-extractor-vmqlc" Jun 02 01:43:38.289767 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:38.289717 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/62166777-842d-49dc-9215-7b408dff79c9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vmqlc\" (UID: \"62166777-842d-49dc-9215-7b408dff79c9\") " pod="openshift-insights/insights-runtime-extractor-vmqlc" Jun 02 01:43:38.390340 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:38.390305 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/62166777-842d-49dc-9215-7b408dff79c9-crio-socket\") pod \"insights-runtime-extractor-vmqlc\" (UID: \"62166777-842d-49dc-9215-7b408dff79c9\") " pod="openshift-insights/insights-runtime-extractor-vmqlc" Jun 02 01:43:38.390340 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:38.390344 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ebbbf316-4688-4404-b302-8cc241ec36b7-registry-tls\") pod \"image-registry-657f6fdbb8-cntl6\" (UID: \"ebbbf316-4688-4404-b302-8cc241ec36b7\") " pod="openshift-image-registry/image-registry-657f6fdbb8-cntl6" Jun 02 01:43:38.390556 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:38.390375 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ebbbf316-4688-4404-b302-8cc241ec36b7-registry-certificates\") pod \"image-registry-657f6fdbb8-cntl6\" (UID: \"ebbbf316-4688-4404-b302-8cc241ec36b7\") " pod="openshift-image-registry/image-registry-657f6fdbb8-cntl6" Jun 02 01:43:38.390556 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:38.390429 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ebbbf316-4688-4404-b302-8cc241ec36b7-bound-sa-token\") pod \"image-registry-657f6fdbb8-cntl6\" (UID: \"ebbbf316-4688-4404-b302-8cc241ec36b7\") " pod="openshift-image-registry/image-registry-657f6fdbb8-cntl6" Jun 02 01:43:38.390556 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:38.390437 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/62166777-842d-49dc-9215-7b408dff79c9-crio-socket\") pod \"insights-runtime-extractor-vmqlc\" (UID: \"62166777-842d-49dc-9215-7b408dff79c9\") " pod="openshift-insights/insights-runtime-extractor-vmqlc" Jun 02 01:43:38.390556 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:38.390468 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/62166777-842d-49dc-9215-7b408dff79c9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vmqlc\" (UID: \"62166777-842d-49dc-9215-7b408dff79c9\") " pod="openshift-insights/insights-runtime-extractor-vmqlc" Jun 02 01:43:38.390556 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:38.390501 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j85fk\" (UniqueName: \"kubernetes.io/projected/ebbbf316-4688-4404-b302-8cc241ec36b7-kube-api-access-j85fk\") pod \"image-registry-657f6fdbb8-cntl6\" (UID: \"ebbbf316-4688-4404-b302-8cc241ec36b7\") " pod="openshift-image-registry/image-registry-657f6fdbb8-cntl6" Jun 02 01:43:38.390556 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:38.390535 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/62166777-842d-49dc-9215-7b408dff79c9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vmqlc\" (UID: \"62166777-842d-49dc-9215-7b408dff79c9\") " pod="openshift-insights/insights-runtime-extractor-vmqlc" Jun 02 01:43:38.390824 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:38.390561 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ebbbf316-4688-4404-b302-8cc241ec36b7-trusted-ca\") pod \"image-registry-657f6fdbb8-cntl6\" (UID: \"ebbbf316-4688-4404-b302-8cc241ec36b7\") " pod="openshift-image-registry/image-registry-657f6fdbb8-cntl6" Jun 02 01:43:38.390824 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:38.390593 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-v245p\" (UniqueName: \"kubernetes.io/projected/62166777-842d-49dc-9215-7b408dff79c9-kube-api-access-v245p\") pod \"insights-runtime-extractor-vmqlc\" (UID: \"62166777-842d-49dc-9215-7b408dff79c9\") " pod="openshift-insights/insights-runtime-extractor-vmqlc" Jun 02 01:43:38.390824 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:38.390618 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ebbbf316-4688-4404-b302-8cc241ec36b7-ca-trust-extracted\") pod \"image-registry-657f6fdbb8-cntl6\" (UID: \"ebbbf316-4688-4404-b302-8cc241ec36b7\") " pod="openshift-image-registry/image-registry-657f6fdbb8-cntl6" Jun 02 01:43:38.390824 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:38.390675 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ebbbf316-4688-4404-b302-8cc241ec36b7-image-registry-private-configuration\") pod \"image-registry-657f6fdbb8-cntl6\" (UID: \"ebbbf316-4688-4404-b302-8cc241ec36b7\") " pod="openshift-image-registry/image-registry-657f6fdbb8-cntl6" Jun 02 01:43:38.390824 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:38.390712 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ebbbf316-4688-4404-b302-8cc241ec36b7-installation-pull-secrets\") pod \"image-registry-657f6fdbb8-cntl6\" (UID: \"ebbbf316-4688-4404-b302-8cc241ec36b7\") " pod="openshift-image-registry/image-registry-657f6fdbb8-cntl6" Jun 02 01:43:38.390824 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:38.390745 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/62166777-842d-49dc-9215-7b408dff79c9-data-volume\") pod \"insights-runtime-extractor-vmqlc\" (UID: \"62166777-842d-49dc-9215-7b408dff79c9\") " pod="openshift-insights/insights-runtime-extractor-vmqlc" Jun 02 01:43:38.391061 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:38.390997 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/62166777-842d-49dc-9215-7b408dff79c9-data-volume\") pod \"insights-runtime-extractor-vmqlc\" (UID: \"62166777-842d-49dc-9215-7b408dff79c9\") " pod="openshift-insights/insights-runtime-extractor-vmqlc" Jun 02 01:43:38.391124 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:38.391107 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/62166777-842d-49dc-9215-7b408dff79c9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vmqlc\" (UID: \"62166777-842d-49dc-9215-7b408dff79c9\") " pod="openshift-insights/insights-runtime-extractor-vmqlc" Jun 02 01:43:38.393145 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:38.393117 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/62166777-842d-49dc-9215-7b408dff79c9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vmqlc\" (UID: \"62166777-842d-49dc-9215-7b408dff79c9\") " pod="openshift-insights/insights-runtime-extractor-vmqlc" Jun 02 01:43:38.401988 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:38.401966 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v245p\" (UniqueName: \"kubernetes.io/projected/62166777-842d-49dc-9215-7b408dff79c9-kube-api-access-v245p\") pod \"insights-runtime-extractor-vmqlc\" (UID: \"62166777-842d-49dc-9215-7b408dff79c9\") " pod="openshift-insights/insights-runtime-extractor-vmqlc" Jun 02 01:43:38.491491 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:38.491410 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ebbbf316-4688-4404-b302-8cc241ec36b7-registry-certificates\") pod \"image-registry-657f6fdbb8-cntl6\" (UID: \"ebbbf316-4688-4404-b302-8cc241ec36b7\") " pod="openshift-image-registry/image-registry-657f6fdbb8-cntl6" Jun 02 01:43:38.491491 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:38.491446 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ebbbf316-4688-4404-b302-8cc241ec36b7-bound-sa-token\") pod \"image-registry-657f6fdbb8-cntl6\" (UID: \"ebbbf316-4688-4404-b302-8cc241ec36b7\") " pod="openshift-image-registry/image-registry-657f6fdbb8-cntl6" Jun 02 01:43:38.491491 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:38.491477 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-j85fk\" (UniqueName: \"kubernetes.io/projected/ebbbf316-4688-4404-b302-8cc241ec36b7-kube-api-access-j85fk\") pod \"image-registry-657f6fdbb8-cntl6\" (UID: \"ebbbf316-4688-4404-b302-8cc241ec36b7\") " pod="openshift-image-registry/image-registry-657f6fdbb8-cntl6" Jun 02 01:43:38.491727 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:38.491635 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ebbbf316-4688-4404-b302-8cc241ec36b7-trusted-ca\") pod \"image-registry-657f6fdbb8-cntl6\" (UID: \"ebbbf316-4688-4404-b302-8cc241ec36b7\") " pod="openshift-image-registry/image-registry-657f6fdbb8-cntl6" Jun 02 01:43:38.491727 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:38.491676 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ebbbf316-4688-4404-b302-8cc241ec36b7-ca-trust-extracted\") pod \"image-registry-657f6fdbb8-cntl6\" (UID: \"ebbbf316-4688-4404-b302-8cc241ec36b7\") " pod="openshift-image-registry/image-registry-657f6fdbb8-cntl6" Jun 02 01:43:38.491832 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:38.491736 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ebbbf316-4688-4404-b302-8cc241ec36b7-image-registry-private-configuration\") pod \"image-registry-657f6fdbb8-cntl6\" (UID: \"ebbbf316-4688-4404-b302-8cc241ec36b7\") " pod="openshift-image-registry/image-registry-657f6fdbb8-cntl6" Jun 02 01:43:38.491832 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:38.491781 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ebbbf316-4688-4404-b302-8cc241ec36b7-installation-pull-secrets\") pod \"image-registry-657f6fdbb8-cntl6\" (UID: \"ebbbf316-4688-4404-b302-8cc241ec36b7\") " pod="openshift-image-registry/image-registry-657f6fdbb8-cntl6" Jun 02 01:43:38.491931 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:38.491834 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ebbbf316-4688-4404-b302-8cc241ec36b7-registry-tls\") pod \"image-registry-657f6fdbb8-cntl6\" (UID: \"ebbbf316-4688-4404-b302-8cc241ec36b7\") " pod="openshift-image-registry/image-registry-657f6fdbb8-cntl6" Jun 02 01:43:38.492174 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:38.492152 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ebbbf316-4688-4404-b302-8cc241ec36b7-ca-trust-extracted\") pod \"image-registry-657f6fdbb8-cntl6\" (UID: \"ebbbf316-4688-4404-b302-8cc241ec36b7\") " pod="openshift-image-registry/image-registry-657f6fdbb8-cntl6" Jun 02 01:43:38.492460 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:38.492424 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ebbbf316-4688-4404-b302-8cc241ec36b7-registry-certificates\") pod \"image-registry-657f6fdbb8-cntl6\" (UID: \"ebbbf316-4688-4404-b302-8cc241ec36b7\") " pod="openshift-image-registry/image-registry-657f6fdbb8-cntl6" Jun 02 01:43:38.492897 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:38.492870 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ebbbf316-4688-4404-b302-8cc241ec36b7-trusted-ca\") pod \"image-registry-657f6fdbb8-cntl6\" (UID: \"ebbbf316-4688-4404-b302-8cc241ec36b7\") " pod="openshift-image-registry/image-registry-657f6fdbb8-cntl6" Jun 02 01:43:38.494368 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:38.494344 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ebbbf316-4688-4404-b302-8cc241ec36b7-installation-pull-secrets\") pod \"image-registry-657f6fdbb8-cntl6\" (UID: \"ebbbf316-4688-4404-b302-8cc241ec36b7\") " pod="openshift-image-registry/image-registry-657f6fdbb8-cntl6" Jun 02 01:43:38.494368 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:38.494346 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ebbbf316-4688-4404-b302-8cc241ec36b7-image-registry-private-configuration\") pod \"image-registry-657f6fdbb8-cntl6\" (UID: \"ebbbf316-4688-4404-b302-8cc241ec36b7\") " pod="openshift-image-registry/image-registry-657f6fdbb8-cntl6" Jun 02 01:43:38.494543 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:38.494527 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ebbbf316-4688-4404-b302-8cc241ec36b7-registry-tls\") pod \"image-registry-657f6fdbb8-cntl6\" (UID: \"ebbbf316-4688-4404-b302-8cc241ec36b7\") " pod="openshift-image-registry/image-registry-657f6fdbb8-cntl6" Jun 02 01:43:38.497585 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:38.497567 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vmqlc" Jun 02 01:43:38.499237 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:38.499216 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ebbbf316-4688-4404-b302-8cc241ec36b7-bound-sa-token\") pod \"image-registry-657f6fdbb8-cntl6\" (UID: \"ebbbf316-4688-4404-b302-8cc241ec36b7\") " pod="openshift-image-registry/image-registry-657f6fdbb8-cntl6" Jun 02 01:43:38.499746 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:38.499722 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j85fk\" (UniqueName: \"kubernetes.io/projected/ebbbf316-4688-4404-b302-8cc241ec36b7-kube-api-access-j85fk\") pod \"image-registry-657f6fdbb8-cntl6\" (UID: \"ebbbf316-4688-4404-b302-8cc241ec36b7\") " pod="openshift-image-registry/image-registry-657f6fdbb8-cntl6" Jun 02 01:43:38.541100 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:38.541010 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-657f6fdbb8-cntl6" Jun 02 01:43:38.639035 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:38.639001 2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vmqlc"] Jun 02 01:43:38.642020 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:43:38.641988 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62166777_842d_49dc_9215_7b408dff79c9.slice/crio-542f06a3b32e8c4a2688d02fd1c8d3a8709c01f156f76483fdc0db5b190b9449 WatchSource:0}: Error finding container 542f06a3b32e8c4a2688d02fd1c8d3a8709c01f156f76483fdc0db5b190b9449: Status 404 returned error can't find the container with id 542f06a3b32e8c4a2688d02fd1c8d3a8709c01f156f76483fdc0db5b190b9449 Jun 02 01:43:38.680706 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:38.680680 2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-657f6fdbb8-cntl6"] Jun 02 01:43:38.687026 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:43:38.687001 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebbbf316_4688_4404_b302_8cc241ec36b7.slice/crio-eaa25e3530ace398c8bbb0b6b77bedfccf0c15f4817acf18157dc70ca6709703 WatchSource:0}: Error finding container eaa25e3530ace398c8bbb0b6b77bedfccf0c15f4817acf18157dc70ca6709703: Status 404 returned error can't find the container with id eaa25e3530ace398c8bbb0b6b77bedfccf0c15f4817acf18157dc70ca6709703 Jun 02 01:43:39.357306 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:39.357275 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vmqlc" event={"ID":"62166777-842d-49dc-9215-7b408dff79c9","Type":"ContainerStarted","Data":"0eb0a3d4e142cc8f11484a5bd4ca38c1bb197046c4fc2d5c6c56a1486addcf20"} Jun 02 01:43:39.357667 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:39.357314 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vmqlc" event={"ID":"62166777-842d-49dc-9215-7b408dff79c9","Type":"ContainerStarted","Data":"e73f285e1fbd9e1c8fbbd97d157cd67b017563a796341ac3946c9c06589613ff"} Jun 02 01:43:39.357667 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:39.357329 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vmqlc" event={"ID":"62166777-842d-49dc-9215-7b408dff79c9","Type":"ContainerStarted","Data":"542f06a3b32e8c4a2688d02fd1c8d3a8709c01f156f76483fdc0db5b190b9449"} Jun 02 01:43:39.358621 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:39.358596 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-657f6fdbb8-cntl6" event={"ID":"ebbbf316-4688-4404-b302-8cc241ec36b7","Type":"ContainerStarted","Data":"3f956a230acd8b5cdc13dd0c5121241f975657038400b4f81b7d31e689e1e1a2"} Jun 02 01:43:39.358701 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:39.358631 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-657f6fdbb8-cntl6" event={"ID":"ebbbf316-4688-4404-b302-8cc241ec36b7","Type":"ContainerStarted","Data":"eaa25e3530ace398c8bbb0b6b77bedfccf0c15f4817acf18157dc70ca6709703"} Jun 02 01:43:39.358785 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:39.358764 2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-657f6fdbb8-cntl6" Jun 02 01:43:39.376372 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:39.376330 2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/image-registry-657f6fdbb8-cntl6" podStartSLOduration=1.376318179 podStartE2EDuration="1.376318179s" podCreationTimestamp="2026-06-02 01:43:38 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 01:43:39.375568449 +0000 UTC m=+137.021375093" watchObservedRunningTime="2026-06-02 01:43:39.376318179 +0000 UTC m=+137.022124822" Jun 02 01:43:41.365363 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:41.365274 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vmqlc" event={"ID":"62166777-842d-49dc-9215-7b408dff79c9","Type":"ContainerStarted","Data":"bff386fb5f53316fee2dc537ccb45a099d459220806bec9f37473ee6ea028e73"} Jun 02 01:43:41.382567 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:41.382515 2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-vmqlc" podStartSLOduration=1.160179673 podStartE2EDuration="3.382502585s" podCreationTimestamp="2026-06-02 01:43:38 +0000 UTC" firstStartedPulling="2026-06-02 01:43:38.710103737 +0000 UTC m=+136.355910370" lastFinishedPulling="2026-06-02 01:43:40.932426649 +0000 UTC m=+138.578233282" observedRunningTime="2026-06-02 01:43:41.380952514 +0000 UTC m=+139.026759156" watchObservedRunningTime="2026-06-02 01:43:41.382502585 +0000 UTC m=+139.028309226" Jun 02 01:43:58.545447 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:58.545413 2571 patch_prober.go:28] interesting pod/image-registry-657f6fdbb8-cntl6 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jun 02 01:43:58.545875 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:58.545476 2571 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-657f6fdbb8-cntl6" podUID="ebbbf316-4688-4404-b302-8cc241ec36b7" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jun 02 01:43:59.252956 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:43:59.252911 2571 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-fs4kq" podUID="97195e1b-54fc-432f-a909-88dac67a6235" Jun 02 01:43:59.257481 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:43:59.257454 2571 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-ingress-canary/ingress-canary-m4mvd" podUID="e5cbb2d6-6afa-4be3-9aab-2f3bd30125bd" Jun 02 01:43:59.404662 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:59.404628 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-m4mvd" Jun 02 01:43:59.404851 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:43:59.404628 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fs4kq" Jun 02 01:44:00.365588 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:00.365555 2571 patch_prober.go:28] interesting pod/image-registry-657f6fdbb8-cntl6 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jun 02 01:44:00.365933 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:00.365610 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-657f6fdbb8-cntl6" podUID="ebbbf316-4688-4404-b302-8cc241ec36b7" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jun 02 01:44:04.177866 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:04.177835 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e5cbb2d6-6afa-4be3-9aab-2f3bd30125bd-cert\") pod \"ingress-canary-m4mvd\" (UID: \"e5cbb2d6-6afa-4be3-9aab-2f3bd30125bd\") " pod="openshift-ingress-canary/ingress-canary-m4mvd" Jun 02 01:44:04.178305 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:04.177893 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/97195e1b-54fc-432f-a909-88dac67a6235-metrics-tls\") pod \"dns-default-fs4kq\" (UID: \"97195e1b-54fc-432f-a909-88dac67a6235\") " pod="openshift-dns/dns-default-fs4kq" Jun 02 01:44:04.180231 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:04.180196 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/97195e1b-54fc-432f-a909-88dac67a6235-metrics-tls\") pod \"dns-default-fs4kq\" (UID: \"97195e1b-54fc-432f-a909-88dac67a6235\") " pod="openshift-dns/dns-default-fs4kq" Jun 02 01:44:04.180336 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:04.180274 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e5cbb2d6-6afa-4be3-9aab-2f3bd30125bd-cert\") pod \"ingress-canary-m4mvd\" (UID: \"e5cbb2d6-6afa-4be3-9aab-2f3bd30125bd\") " pod="openshift-ingress-canary/ingress-canary-m4mvd" Jun 02 01:44:04.208559 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:04.208536 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-4kzbv\"" Jun 02 01:44:04.208698 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:04.208538 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-q2kdm\"" Jun 02 01:44:04.215602 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:04.215584 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-m4mvd" Jun 02 01:44:04.215702 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:04.215605 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fs4kq" Jun 02 01:44:04.304174 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:04.304028 2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-4rfv6"] Jun 02 01:44:04.314767 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:04.313417 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4rfv6" Jun 02 01:44:04.316505 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:04.316326 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jun 02 01:44:04.316505 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:04.316353 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-76pc2\"" Jun 02 01:44:04.317228 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:04.316983 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jun 02 01:44:04.318071 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:04.317360 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jun 02 01:44:04.318071 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:04.317452 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jun 02 01:44:04.318071 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:04.317702 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jun 02 01:44:04.318071 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:04.317895 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jun 02 01:44:04.379895 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:04.379754 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/60142a30-6a11-4fca-9e87-30271db2c7f6-node-exporter-textfile\") pod \"node-exporter-4rfv6\" (UID: \"60142a30-6a11-4fca-9e87-30271db2c7f6\") " pod="openshift-monitoring/node-exporter-4rfv6" Jun 02 01:44:04.379895 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:04.379810 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/60142a30-6a11-4fca-9e87-30271db2c7f6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4rfv6\" (UID: \"60142a30-6a11-4fca-9e87-30271db2c7f6\") " pod="openshift-monitoring/node-exporter-4rfv6" Jun 02 01:44:04.379895 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:04.379854 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/60142a30-6a11-4fca-9e87-30271db2c7f6-node-exporter-accelerators-collector-config\") pod \"node-exporter-4rfv6\" (UID: \"60142a30-6a11-4fca-9e87-30271db2c7f6\") " pod="openshift-monitoring/node-exporter-4rfv6" Jun 02 01:44:04.379895 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:04.379894 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60142a30-6a11-4fca-9e87-30271db2c7f6-metrics-client-ca\") pod \"node-exporter-4rfv6\" (UID: \"60142a30-6a11-4fca-9e87-30271db2c7f6\") " pod="openshift-monitoring/node-exporter-4rfv6" Jun 02 01:44:04.380249 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:04.379981 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/60142a30-6a11-4fca-9e87-30271db2c7f6-root\") pod \"node-exporter-4rfv6\" (UID: \"60142a30-6a11-4fca-9e87-30271db2c7f6\") " pod="openshift-monitoring/node-exporter-4rfv6" Jun 02 01:44:04.380249 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:04.380017 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/60142a30-6a11-4fca-9e87-30271db2c7f6-sys\") pod \"node-exporter-4rfv6\" (UID: \"60142a30-6a11-4fca-9e87-30271db2c7f6\") " pod="openshift-monitoring/node-exporter-4rfv6" Jun 02 01:44:04.380249 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:04.380049 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jlqd\" (UniqueName: \"kubernetes.io/projected/60142a30-6a11-4fca-9e87-30271db2c7f6-kube-api-access-7jlqd\") pod \"node-exporter-4rfv6\" (UID: \"60142a30-6a11-4fca-9e87-30271db2c7f6\") " pod="openshift-monitoring/node-exporter-4rfv6" Jun 02 01:44:04.380249 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:04.380092 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/60142a30-6a11-4fca-9e87-30271db2c7f6-node-exporter-tls\") pod \"node-exporter-4rfv6\" (UID: \"60142a30-6a11-4fca-9e87-30271db2c7f6\") " pod="openshift-monitoring/node-exporter-4rfv6" Jun 02 01:44:04.380249 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:04.380145 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/60142a30-6a11-4fca-9e87-30271db2c7f6-node-exporter-wtmp\") pod \"node-exporter-4rfv6\" (UID: \"60142a30-6a11-4fca-9e87-30271db2c7f6\") " pod="openshift-monitoring/node-exporter-4rfv6" Jun 02 01:44:04.480552 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:04.480455 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/60142a30-6a11-4fca-9e87-30271db2c7f6-node-exporter-accelerators-collector-config\") pod \"node-exporter-4rfv6\" (UID: \"60142a30-6a11-4fca-9e87-30271db2c7f6\") " pod="openshift-monitoring/node-exporter-4rfv6" Jun 02 01:44:04.480552 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:04.480509 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60142a30-6a11-4fca-9e87-30271db2c7f6-metrics-client-ca\") pod \"node-exporter-4rfv6\" (UID: \"60142a30-6a11-4fca-9e87-30271db2c7f6\") " pod="openshift-monitoring/node-exporter-4rfv6" Jun 02 01:44:04.480552 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:04.480537 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/60142a30-6a11-4fca-9e87-30271db2c7f6-root\") pod \"node-exporter-4rfv6\" (UID: \"60142a30-6a11-4fca-9e87-30271db2c7f6\") " pod="openshift-monitoring/node-exporter-4rfv6" Jun 02 01:44:04.480825 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:04.480564 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/60142a30-6a11-4fca-9e87-30271db2c7f6-sys\") pod \"node-exporter-4rfv6\" (UID: \"60142a30-6a11-4fca-9e87-30271db2c7f6\") " pod="openshift-monitoring/node-exporter-4rfv6" Jun 02 01:44:04.480825 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:04.480588 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7jlqd\" (UniqueName: \"kubernetes.io/projected/60142a30-6a11-4fca-9e87-30271db2c7f6-kube-api-access-7jlqd\") pod \"node-exporter-4rfv6\" (UID: \"60142a30-6a11-4fca-9e87-30271db2c7f6\") " pod="openshift-monitoring/node-exporter-4rfv6" Jun 02 01:44:04.480825 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:04.480612 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/60142a30-6a11-4fca-9e87-30271db2c7f6-node-exporter-tls\") pod \"node-exporter-4rfv6\" (UID: \"60142a30-6a11-4fca-9e87-30271db2c7f6\") " pod="openshift-monitoring/node-exporter-4rfv6" Jun 02 01:44:04.480825 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:04.480636 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/60142a30-6a11-4fca-9e87-30271db2c7f6-node-exporter-wtmp\") pod \"node-exporter-4rfv6\" (UID: \"60142a30-6a11-4fca-9e87-30271db2c7f6\") " pod="openshift-monitoring/node-exporter-4rfv6" Jun 02 01:44:04.480825 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:04.480657 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/60142a30-6a11-4fca-9e87-30271db2c7f6-root\") pod \"node-exporter-4rfv6\" (UID: \"60142a30-6a11-4fca-9e87-30271db2c7f6\") " pod="openshift-monitoring/node-exporter-4rfv6" Jun 02 01:44:04.480825 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:04.480679 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/60142a30-6a11-4fca-9e87-30271db2c7f6-sys\") pod \"node-exporter-4rfv6\" (UID: \"60142a30-6a11-4fca-9e87-30271db2c7f6\") " pod="openshift-monitoring/node-exporter-4rfv6" Jun 02 01:44:04.480825 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:04.480670 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/60142a30-6a11-4fca-9e87-30271db2c7f6-node-exporter-textfile\") pod \"node-exporter-4rfv6\" (UID: \"60142a30-6a11-4fca-9e87-30271db2c7f6\") " pod="openshift-monitoring/node-exporter-4rfv6" Jun 02 01:44:04.480825 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:04.480744 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/60142a30-6a11-4fca-9e87-30271db2c7f6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4rfv6\" (UID: \"60142a30-6a11-4fca-9e87-30271db2c7f6\") " pod="openshift-monitoring/node-exporter-4rfv6" Jun 02 01:44:04.480825 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:44:04.480775 2571 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jun 02 01:44:04.481293 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:44:04.480834 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60142a30-6a11-4fca-9e87-30271db2c7f6-node-exporter-tls podName:60142a30-6a11-4fca-9e87-30271db2c7f6 nodeName:}" failed. No retries permitted until 2026-06-02 01:44:04.980813575 +0000 UTC m=+162.626620201 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/60142a30-6a11-4fca-9e87-30271db2c7f6-node-exporter-tls") pod "node-exporter-4rfv6" (UID: "60142a30-6a11-4fca-9e87-30271db2c7f6") : secret "node-exporter-tls" not found Jun 02 01:44:04.481293 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:04.481046 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/60142a30-6a11-4fca-9e87-30271db2c7f6-node-exporter-wtmp\") pod \"node-exporter-4rfv6\" (UID: \"60142a30-6a11-4fca-9e87-30271db2c7f6\") " pod="openshift-monitoring/node-exporter-4rfv6" Jun 02 01:44:04.484164 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:04.481597 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/60142a30-6a11-4fca-9e87-30271db2c7f6-node-exporter-textfile\") pod \"node-exporter-4rfv6\" (UID: \"60142a30-6a11-4fca-9e87-30271db2c7f6\") " pod="openshift-monitoring/node-exporter-4rfv6" Jun 02 01:44:04.484164 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:04.481657 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/60142a30-6a11-4fca-9e87-30271db2c7f6-node-exporter-accelerators-collector-config\") pod \"node-exporter-4rfv6\" (UID: \"60142a30-6a11-4fca-9e87-30271db2c7f6\") " pod="openshift-monitoring/node-exporter-4rfv6" Jun 02 01:44:04.484164 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:04.481758 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60142a30-6a11-4fca-9e87-30271db2c7f6-metrics-client-ca\") pod \"node-exporter-4rfv6\" (UID: \"60142a30-6a11-4fca-9e87-30271db2c7f6\") " pod="openshift-monitoring/node-exporter-4rfv6" Jun 02 01:44:04.487935 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:04.487909 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/60142a30-6a11-4fca-9e87-30271db2c7f6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4rfv6\" (UID: \"60142a30-6a11-4fca-9e87-30271db2c7f6\") " pod="openshift-monitoring/node-exporter-4rfv6" Jun 02 01:44:04.492156 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:04.492117 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jlqd\" (UniqueName: \"kubernetes.io/projected/60142a30-6a11-4fca-9e87-30271db2c7f6-kube-api-access-7jlqd\") pod \"node-exporter-4rfv6\" (UID: \"60142a30-6a11-4fca-9e87-30271db2c7f6\") " pod="openshift-monitoring/node-exporter-4rfv6" Jun 02 01:44:04.596960 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:04.596934 2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-m4mvd"] Jun 02 01:44:04.600198 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:04.600125 2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fs4kq"] Jun 02 01:44:04.600324 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:44:04.600239 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5cbb2d6_6afa_4be3_9aab_2f3bd30125bd.slice/crio-0dbc1f1400ea0cfc6333476db3b8bbb365cd94d2681daae5c040a130a93dbcc5 WatchSource:0}: Error finding container 0dbc1f1400ea0cfc6333476db3b8bbb365cd94d2681daae5c040a130a93dbcc5: Status 404 returned error can't find the container with id 0dbc1f1400ea0cfc6333476db3b8bbb365cd94d2681daae5c040a130a93dbcc5 Jun 02 01:44:04.603073 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:44:04.603047 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97195e1b_54fc_432f_a909_88dac67a6235.slice/crio-dae673c7b4d2cd6f8ca1d1099765a121277fa394ecb5135eede315209ed74e62 WatchSource:0}: Error finding container dae673c7b4d2cd6f8ca1d1099765a121277fa394ecb5135eede315209ed74e62: Status 404 returned error can't find the container with id dae673c7b4d2cd6f8ca1d1099765a121277fa394ecb5135eede315209ed74e62 Jun 02 01:44:04.985194 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:04.985152 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/60142a30-6a11-4fca-9e87-30271db2c7f6-node-exporter-tls\") pod \"node-exporter-4rfv6\" (UID: \"60142a30-6a11-4fca-9e87-30271db2c7f6\") " pod="openshift-monitoring/node-exporter-4rfv6" Jun 02 01:44:04.987940 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:04.987912 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/60142a30-6a11-4fca-9e87-30271db2c7f6-node-exporter-tls\") pod \"node-exporter-4rfv6\" (UID: \"60142a30-6a11-4fca-9e87-30271db2c7f6\") " pod="openshift-monitoring/node-exporter-4rfv6" Jun 02 01:44:05.226576 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:05.226530 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4rfv6" Jun 02 01:44:05.238170 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:44:05.237608 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60142a30_6a11_4fca_9e87_30271db2c7f6.slice/crio-2fcee9f67b25f2dd887fd2b81f09d2c57c962a79d05d819d63619020c98d94c8 WatchSource:0}: Error finding container 2fcee9f67b25f2dd887fd2b81f09d2c57c962a79d05d819d63619020c98d94c8: Status 404 returned error can't find the container with id 2fcee9f67b25f2dd887fd2b81f09d2c57c962a79d05d819d63619020c98d94c8 Jun 02 01:44:05.346800 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:05.346765 2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jun 02 01:44:05.351836 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:05.351808 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jun 02 01:44:05.354690 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:05.354409 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jun 02 01:44:05.354690 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:05.354600 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jun 02 01:44:05.354690 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:05.354684 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-4ftw9\"" Jun 02 01:44:05.354920 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:05.354842 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jun 02 01:44:05.355471 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:05.355019 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jun 02 01:44:05.355471 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:05.355196 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jun 02 01:44:05.355471 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:05.355276 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jun 02 01:44:05.355471 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:05.355367 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jun 02 01:44:05.355705 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:05.355545 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jun 02 01:44:05.355828 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:05.355769 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jun 02 01:44:05.363331 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:05.363306 2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jun 02 01:44:05.389859 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:05.389830 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7f4e3de2-1d34-464b-8dc3-dd008db7de61-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7f4e3de2-1d34-464b-8dc3-dd008db7de61\") " pod="openshift-monitoring/alertmanager-main-0" Jun 02 01:44:05.389994 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:05.389876 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmcqh\" (UniqueName: \"kubernetes.io/projected/7f4e3de2-1d34-464b-8dc3-dd008db7de61-kube-api-access-wmcqh\") pod \"alertmanager-main-0\" (UID: \"7f4e3de2-1d34-464b-8dc3-dd008db7de61\") " pod="openshift-monitoring/alertmanager-main-0" Jun 02 01:44:05.389994 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:05.389907 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7f4e3de2-1d34-464b-8dc3-dd008db7de61-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7f4e3de2-1d34-464b-8dc3-dd008db7de61\") " pod="openshift-monitoring/alertmanager-main-0" Jun 02 01:44:05.389994 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:05.389938 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7f4e3de2-1d34-464b-8dc3-dd008db7de61-config-out\") pod \"alertmanager-main-0\" (UID: \"7f4e3de2-1d34-464b-8dc3-dd008db7de61\") " pod="openshift-monitoring/alertmanager-main-0" Jun 02 01:44:05.389994 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:05.389973 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f4e3de2-1d34-464b-8dc3-dd008db7de61-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7f4e3de2-1d34-464b-8dc3-dd008db7de61\") " pod="openshift-monitoring/alertmanager-main-0" Jun 02 01:44:05.390509 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:05.390012 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7f4e3de2-1d34-464b-8dc3-dd008db7de61-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7f4e3de2-1d34-464b-8dc3-dd008db7de61\") " pod="openshift-monitoring/alertmanager-main-0" Jun 02 01:44:05.390509 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:05.390038 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7f4e3de2-1d34-464b-8dc3-dd008db7de61-web-config\") pod \"alertmanager-main-0\" (UID: \"7f4e3de2-1d34-464b-8dc3-dd008db7de61\") " pod="openshift-monitoring/alertmanager-main-0" Jun 02 01:44:05.390509 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:05.390066 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7f4e3de2-1d34-464b-8dc3-dd008db7de61-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7f4e3de2-1d34-464b-8dc3-dd008db7de61\") " pod="openshift-monitoring/alertmanager-main-0" Jun 02 01:44:05.390509 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:05.390089 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7f4e3de2-1d34-464b-8dc3-dd008db7de61-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7f4e3de2-1d34-464b-8dc3-dd008db7de61\") " pod="openshift-monitoring/alertmanager-main-0" Jun 02 01:44:05.390509 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:05.390153 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7f4e3de2-1d34-464b-8dc3-dd008db7de61-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7f4e3de2-1d34-464b-8dc3-dd008db7de61\") " pod="openshift-monitoring/alertmanager-main-0" Jun 02 01:44:05.390509 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:05.390216 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7f4e3de2-1d34-464b-8dc3-dd008db7de61-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7f4e3de2-1d34-464b-8dc3-dd008db7de61\") " pod="openshift-monitoring/alertmanager-main-0" Jun 02 01:44:05.390509 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:05.390264 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7f4e3de2-1d34-464b-8dc3-dd008db7de61-config-volume\") pod \"alertmanager-main-0\" (UID: \"7f4e3de2-1d34-464b-8dc3-dd008db7de61\") " pod="openshift-monitoring/alertmanager-main-0" Jun 02 01:44:05.390509 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:05.390296 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7f4e3de2-1d34-464b-8dc3-dd008db7de61-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7f4e3de2-1d34-464b-8dc3-dd008db7de61\") " pod="openshift-monitoring/alertmanager-main-0" Jun 02 01:44:05.419835 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:05.419799 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-m4mvd" event={"ID":"e5cbb2d6-6afa-4be3-9aab-2f3bd30125bd","Type":"ContainerStarted","Data":"0dbc1f1400ea0cfc6333476db3b8bbb365cd94d2681daae5c040a130a93dbcc5"} Jun 02 01:44:05.421226 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:05.421199 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4rfv6" event={"ID":"60142a30-6a11-4fca-9e87-30271db2c7f6","Type":"ContainerStarted","Data":"2fcee9f67b25f2dd887fd2b81f09d2c57c962a79d05d819d63619020c98d94c8"} Jun 02 01:44:05.422399 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:05.422373 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fs4kq" event={"ID":"97195e1b-54fc-432f-a909-88dac67a6235","Type":"ContainerStarted","Data":"dae673c7b4d2cd6f8ca1d1099765a121277fa394ecb5135eede315209ed74e62"} Jun 02 01:44:05.490880 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:05.490805 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7f4e3de2-1d34-464b-8dc3-dd008db7de61-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7f4e3de2-1d34-464b-8dc3-dd008db7de61\") " pod="openshift-monitoring/alertmanager-main-0" Jun 02 01:44:05.490880 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:05.490855 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wmcqh\" (UniqueName: \"kubernetes.io/projected/7f4e3de2-1d34-464b-8dc3-dd008db7de61-kube-api-access-wmcqh\") pod \"alertmanager-main-0\" (UID: \"7f4e3de2-1d34-464b-8dc3-dd008db7de61\") " pod="openshift-monitoring/alertmanager-main-0" Jun 02 01:44:05.491087 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:05.490887 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7f4e3de2-1d34-464b-8dc3-dd008db7de61-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7f4e3de2-1d34-464b-8dc3-dd008db7de61\") " pod="openshift-monitoring/alertmanager-main-0" Jun 02 01:44:05.491087 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:05.490916 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7f4e3de2-1d34-464b-8dc3-dd008db7de61-config-out\") pod \"alertmanager-main-0\" (UID: \"7f4e3de2-1d34-464b-8dc3-dd008db7de61\") " pod="openshift-monitoring/alertmanager-main-0" Jun 02 01:44:05.491087 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:05.490951 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f4e3de2-1d34-464b-8dc3-dd008db7de61-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7f4e3de2-1d34-464b-8dc3-dd008db7de61\") " pod="openshift-monitoring/alertmanager-main-0" Jun 02 01:44:05.491087 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:05.490973 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7f4e3de2-1d34-464b-8dc3-dd008db7de61-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7f4e3de2-1d34-464b-8dc3-dd008db7de61\") " pod="openshift-monitoring/alertmanager-main-0" Jun 02 01:44:05.491087 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:05.491002 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7f4e3de2-1d34-464b-8dc3-dd008db7de61-web-config\") pod \"alertmanager-main-0\" (UID: \"7f4e3de2-1d34-464b-8dc3-dd008db7de61\") " pod="openshift-monitoring/alertmanager-main-0" Jun 02 01:44:05.491087 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:05.491026 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7f4e3de2-1d34-464b-8dc3-dd008db7de61-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7f4e3de2-1d34-464b-8dc3-dd008db7de61\") " pod="openshift-monitoring/alertmanager-main-0" Jun 02 01:44:05.491087 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:05.491047 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7f4e3de2-1d34-464b-8dc3-dd008db7de61-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7f4e3de2-1d34-464b-8dc3-dd008db7de61\") " pod="openshift-monitoring/alertmanager-main-0" Jun 02 01:44:05.491087 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:05.491078 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7f4e3de2-1d34-464b-8dc3-dd008db7de61-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7f4e3de2-1d34-464b-8dc3-dd008db7de61\") " pod="openshift-monitoring/alertmanager-main-0" Jun 02 01:44:05.491515 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:05.491109 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7f4e3de2-1d34-464b-8dc3-dd008db7de61-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7f4e3de2-1d34-464b-8dc3-dd008db7de61\") " pod="openshift-monitoring/alertmanager-main-0" Jun 02 01:44:05.491515 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:05.491145 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7f4e3de2-1d34-464b-8dc3-dd008db7de61-config-volume\") pod \"alertmanager-main-0\" (UID: \"7f4e3de2-1d34-464b-8dc3-dd008db7de61\") " pod="openshift-monitoring/alertmanager-main-0" Jun 02 01:44:05.491515 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:05.491172 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7f4e3de2-1d34-464b-8dc3-dd008db7de61-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7f4e3de2-1d34-464b-8dc3-dd008db7de61\") " pod="openshift-monitoring/alertmanager-main-0" Jun 02 01:44:05.491515 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:44:05.491439 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7f4e3de2-1d34-464b-8dc3-dd008db7de61-alertmanager-trusted-ca-bundle podName:7f4e3de2-1d34-464b-8dc3-dd008db7de61 nodeName:}" failed. No retries permitted until 2026-06-02 01:44:05.991413016 +0000 UTC m=+163.637219653 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/7f4e3de2-1d34-464b-8dc3-dd008db7de61-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "7f4e3de2-1d34-464b-8dc3-dd008db7de61") : configmap references non-existent config key: ca-bundle.crt Jun 02 01:44:05.492956 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:44:05.492262 2571 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jun 02 01:44:05.492956 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:44:05.492323 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f4e3de2-1d34-464b-8dc3-dd008db7de61-secret-alertmanager-main-tls podName:7f4e3de2-1d34-464b-8dc3-dd008db7de61 nodeName:}" failed. No retries permitted until 2026-06-02 01:44:05.992305466 +0000 UTC m=+163.638112087 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/7f4e3de2-1d34-464b-8dc3-dd008db7de61-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "7f4e3de2-1d34-464b-8dc3-dd008db7de61") : secret "alertmanager-main-tls" not found Jun 02 01:44:05.492956 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:05.492625 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7f4e3de2-1d34-464b-8dc3-dd008db7de61-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7f4e3de2-1d34-464b-8dc3-dd008db7de61\") " pod="openshift-monitoring/alertmanager-main-0" Jun 02 01:44:05.493214 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:05.492971 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7f4e3de2-1d34-464b-8dc3-dd008db7de61-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7f4e3de2-1d34-464b-8dc3-dd008db7de61\") " pod="openshift-monitoring/alertmanager-main-0" Jun 02 01:44:05.497922 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:05.497862 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7f4e3de2-1d34-464b-8dc3-dd008db7de61-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7f4e3de2-1d34-464b-8dc3-dd008db7de61\") " pod="openshift-monitoring/alertmanager-main-0" Jun 02 01:44:05.498606 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:05.498552 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7f4e3de2-1d34-464b-8dc3-dd008db7de61-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7f4e3de2-1d34-464b-8dc3-dd008db7de61\") " pod="openshift-monitoring/alertmanager-main-0" Jun 02 01:44:05.499655 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:05.499630 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7f4e3de2-1d34-464b-8dc3-dd008db7de61-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7f4e3de2-1d34-464b-8dc3-dd008db7de61\") " pod="openshift-monitoring/alertmanager-main-0" Jun 02 01:44:05.500006 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:05.499987 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7f4e3de2-1d34-464b-8dc3-dd008db7de61-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7f4e3de2-1d34-464b-8dc3-dd008db7de61\") " pod="openshift-monitoring/alertmanager-main-0" Jun 02 01:44:05.500358 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:05.500337 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7f4e3de2-1d34-464b-8dc3-dd008db7de61-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7f4e3de2-1d34-464b-8dc3-dd008db7de61\") " pod="openshift-monitoring/alertmanager-main-0" Jun 02 01:44:05.503455 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:05.500496 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7f4e3de2-1d34-464b-8dc3-dd008db7de61-config-out\") pod \"alertmanager-main-0\" (UID: \"7f4e3de2-1d34-464b-8dc3-dd008db7de61\") " pod="openshift-monitoring/alertmanager-main-0" Jun 02 01:44:05.503455 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:05.500808 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7f4e3de2-1d34-464b-8dc3-dd008db7de61-web-config\") pod \"alertmanager-main-0\" (UID: \"7f4e3de2-1d34-464b-8dc3-dd008db7de61\") " pod="openshift-monitoring/alertmanager-main-0" Jun 02 01:44:05.503455 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:05.501289 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7f4e3de2-1d34-464b-8dc3-dd008db7de61-config-volume\") pod \"alertmanager-main-0\" (UID: \"7f4e3de2-1d34-464b-8dc3-dd008db7de61\") " pod="openshift-monitoring/alertmanager-main-0" Jun 02 01:44:05.504286 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:05.504244 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmcqh\" (UniqueName: \"kubernetes.io/projected/7f4e3de2-1d34-464b-8dc3-dd008db7de61-kube-api-access-wmcqh\") pod \"alertmanager-main-0\" (UID: \"7f4e3de2-1d34-464b-8dc3-dd008db7de61\") " pod="openshift-monitoring/alertmanager-main-0" Jun 02 01:44:05.994829 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:05.994786 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7f4e3de2-1d34-464b-8dc3-dd008db7de61-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7f4e3de2-1d34-464b-8dc3-dd008db7de61\") " pod="openshift-monitoring/alertmanager-main-0" Jun 02 01:44:05.995010 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:05.994910 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f4e3de2-1d34-464b-8dc3-dd008db7de61-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7f4e3de2-1d34-464b-8dc3-dd008db7de61\") " pod="openshift-monitoring/alertmanager-main-0" Jun 02 01:44:05.995072 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:44:05.995041 2571 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jun 02 01:44:05.995208 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:44:05.995118 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f4e3de2-1d34-464b-8dc3-dd008db7de61-secret-alertmanager-main-tls podName:7f4e3de2-1d34-464b-8dc3-dd008db7de61 nodeName:}" failed. No retries permitted until 2026-06-02 01:44:06.995098344 +0000 UTC m=+164.640904980 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/7f4e3de2-1d34-464b-8dc3-dd008db7de61-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "7f4e3de2-1d34-464b-8dc3-dd008db7de61") : secret "alertmanager-main-tls" not found Jun 02 01:44:05.995858 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:05.995822 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f4e3de2-1d34-464b-8dc3-dd008db7de61-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7f4e3de2-1d34-464b-8dc3-dd008db7de61\") " pod="openshift-monitoring/alertmanager-main-0" Jun 02 01:44:07.005108 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:07.005024 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7f4e3de2-1d34-464b-8dc3-dd008db7de61-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7f4e3de2-1d34-464b-8dc3-dd008db7de61\") " pod="openshift-monitoring/alertmanager-main-0" Jun 02 01:44:07.007364 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:07.007344 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7f4e3de2-1d34-464b-8dc3-dd008db7de61-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7f4e3de2-1d34-464b-8dc3-dd008db7de61\") " pod="openshift-monitoring/alertmanager-main-0" Jun 02 01:44:07.164495 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:07.164453 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jun 02 01:44:07.297159 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:07.297034 2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jun 02 01:44:07.299231 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:44:07.299203 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f4e3de2_1d34_464b_8dc3_dd008db7de61.slice/crio-b0fdba1bad3a6086c5d46fb2ada2df6729a5ff450cdc1325689ca7215eea422a WatchSource:0}: Error finding container b0fdba1bad3a6086c5d46fb2ada2df6729a5ff450cdc1325689ca7215eea422a: Status 404 returned error can't find the container with id b0fdba1bad3a6086c5d46fb2ada2df6729a5ff450cdc1325689ca7215eea422a Jun 02 01:44:07.428837 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:07.428787 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7f4e3de2-1d34-464b-8dc3-dd008db7de61","Type":"ContainerStarted","Data":"b0fdba1bad3a6086c5d46fb2ada2df6729a5ff450cdc1325689ca7215eea422a"} Jun 02 01:44:07.430179 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:07.430146 2571 generic.go:358] "Generic (PLEG): container finished" podID="60142a30-6a11-4fca-9e87-30271db2c7f6" containerID="907418cf981a94822b9928290427be67c0abfdd923d1a7ffdff24e1d48ea75d9" exitCode=0 Jun 02 01:44:07.430280 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:07.430235 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4rfv6" event={"ID":"60142a30-6a11-4fca-9e87-30271db2c7f6","Type":"ContainerDied","Data":"907418cf981a94822b9928290427be67c0abfdd923d1a7ffdff24e1d48ea75d9"} Jun 02 01:44:07.431990 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:07.431962 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fs4kq" event={"ID":"97195e1b-54fc-432f-a909-88dac67a6235","Type":"ContainerStarted","Data":"ee6077bd276467d4a0aee1162eb85fef881cb5e18dfd537df881b2aba7debb60"} Jun 02 01:44:07.432117 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:07.431996 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fs4kq" event={"ID":"97195e1b-54fc-432f-a909-88dac67a6235","Type":"ContainerStarted","Data":"bb30c7075fe93ddcec26b3966cc0ceda86bb2951d2b58b13495e618fa0938a34"} Jun 02 01:44:07.432117 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:07.432105 2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-fs4kq" Jun 02 01:44:07.433588 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:07.433568 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-m4mvd" event={"ID":"e5cbb2d6-6afa-4be3-9aab-2f3bd30125bd","Type":"ContainerStarted","Data":"4f385492ed33fdd4856474e5f042fb176d7fc9414183313febdf364c98e9688a"} Jun 02 01:44:07.461237 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:07.461195 2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-fs4kq" podStartSLOduration=129.397137269 podStartE2EDuration="2m11.461181872s" podCreationTimestamp="2026-06-02 01:41:56 +0000 UTC" firstStartedPulling="2026-06-02 01:44:04.604856722 +0000 UTC m=+162.250663356" lastFinishedPulling="2026-06-02 01:44:06.668901336 +0000 UTC m=+164.314707959" observedRunningTime="2026-06-02 01:44:07.460458644 +0000 UTC m=+165.106265287" watchObservedRunningTime="2026-06-02 01:44:07.461181872 +0000 UTC m=+165.106988513" Jun 02 01:44:07.475369 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:07.475323 2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-m4mvd" podStartSLOduration=129.402766243 podStartE2EDuration="2m11.47531263s" podCreationTimestamp="2026-06-02 01:41:56 +0000 UTC" firstStartedPulling="2026-06-02 01:44:04.602258866 +0000 UTC m=+162.248065486" lastFinishedPulling="2026-06-02 01:44:06.674805235 +0000 UTC m=+164.320611873" observedRunningTime="2026-06-02 01:44:07.475112784 +0000 UTC m=+165.120919427" watchObservedRunningTime="2026-06-02 01:44:07.47531263 +0000 UTC m=+165.121119300" Jun 02 01:44:08.437603 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:08.437570 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4rfv6" event={"ID":"60142a30-6a11-4fca-9e87-30271db2c7f6","Type":"ContainerStarted","Data":"825472d2192ffd27c5b8cd4704651aa1d828921721c4cd73eb7f5d4b35a25c07"} Jun 02 01:44:08.437889 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:08.437612 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4rfv6" event={"ID":"60142a30-6a11-4fca-9e87-30271db2c7f6","Type":"ContainerStarted","Data":"e1193d9f4236ccab4db134ba46b2e96c30205caab066140e9e9949f94ed2617c"} Jun 02 01:44:08.459375 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:08.459334 2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-4rfv6" podStartSLOduration=2.981565496 podStartE2EDuration="4.459321027s" podCreationTimestamp="2026-06-02 01:44:04 +0000 UTC" firstStartedPulling="2026-06-02 01:44:05.240014935 +0000 UTC m=+162.885821559" lastFinishedPulling="2026-06-02 01:44:06.717770464 +0000 UTC m=+164.363577090" observedRunningTime="2026-06-02 01:44:08.457650557 +0000 UTC m=+166.103457200" watchObservedRunningTime="2026-06-02 01:44:08.459321027 +0000 UTC m=+166.105127680" Jun 02 01:44:08.545602 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:08.545575 2571 patch_prober.go:28] interesting pod/image-registry-657f6fdbb8-cntl6 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jun 02 01:44:08.545689 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:08.545632 2571 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-657f6fdbb8-cntl6" podUID="ebbbf316-4688-4404-b302-8cc241ec36b7" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jun 02 01:44:09.057208 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:09.057175 2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5768fcc9dd-w9b6j"] Jun 02 01:44:09.060083 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:09.060067 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5768fcc9dd-w9b6j" Jun 02 01:44:09.062412 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:09.062376 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jun 02 01:44:09.062472 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:09.062416 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"default-dockercfg-bxw44\"" Jun 02 01:44:09.068795 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:09.068774 2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5768fcc9dd-w9b6j"] Jun 02 01:44:09.121696 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:09.121656 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1b8379d9-608a-4c5b-a377-b4b69afb78b7-monitoring-plugin-cert\") pod \"monitoring-plugin-5768fcc9dd-w9b6j\" (UID: \"1b8379d9-608a-4c5b-a377-b4b69afb78b7\") " pod="openshift-monitoring/monitoring-plugin-5768fcc9dd-w9b6j" Jun 02 01:44:09.222693 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:09.222655 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1b8379d9-608a-4c5b-a377-b4b69afb78b7-monitoring-plugin-cert\") pod \"monitoring-plugin-5768fcc9dd-w9b6j\" (UID: \"1b8379d9-608a-4c5b-a377-b4b69afb78b7\") " pod="openshift-monitoring/monitoring-plugin-5768fcc9dd-w9b6j" Jun 02 01:44:09.222861 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:44:09.222764 2571 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jun 02 01:44:09.222861 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:44:09.222833 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b8379d9-608a-4c5b-a377-b4b69afb78b7-monitoring-plugin-cert podName:1b8379d9-608a-4c5b-a377-b4b69afb78b7 nodeName:}" failed. No retries permitted until 2026-06-02 01:44:09.722817463 +0000 UTC m=+167.368624084 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/1b8379d9-608a-4c5b-a377-b4b69afb78b7-monitoring-plugin-cert") pod "monitoring-plugin-5768fcc9dd-w9b6j" (UID: "1b8379d9-608a-4c5b-a377-b4b69afb78b7") : secret "monitoring-plugin-cert" not found Jun 02 01:44:09.441497 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:09.441460 2571 generic.go:358] "Generic (PLEG): container finished" podID="7f4e3de2-1d34-464b-8dc3-dd008db7de61" containerID="347ce50ecee897ac914f2823394cd03b4f3846e2e9ad23a5cd4c71bea9c12a06" exitCode=0 Jun 02 01:44:09.441881 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:09.441548 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7f4e3de2-1d34-464b-8dc3-dd008db7de61","Type":"ContainerDied","Data":"347ce50ecee897ac914f2823394cd03b4f3846e2e9ad23a5cd4c71bea9c12a06"} Jun 02 01:44:09.727216 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:09.727117 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1b8379d9-608a-4c5b-a377-b4b69afb78b7-monitoring-plugin-cert\") pod \"monitoring-plugin-5768fcc9dd-w9b6j\" (UID: \"1b8379d9-608a-4c5b-a377-b4b69afb78b7\") " pod="openshift-monitoring/monitoring-plugin-5768fcc9dd-w9b6j" Jun 02 01:44:09.729474 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:09.729452 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1b8379d9-608a-4c5b-a377-b4b69afb78b7-monitoring-plugin-cert\") pod \"monitoring-plugin-5768fcc9dd-w9b6j\" (UID: \"1b8379d9-608a-4c5b-a377-b4b69afb78b7\") " pod="openshift-monitoring/monitoring-plugin-5768fcc9dd-w9b6j" Jun 02 01:44:09.969451 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:09.969415 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5768fcc9dd-w9b6j" Jun 02 01:44:10.106049 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:10.106014 2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5768fcc9dd-w9b6j"] Jun 02 01:44:10.109066 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:44:10.109036 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b8379d9_608a_4c5b_a377_b4b69afb78b7.slice/crio-2cb62740cc3397d6ed2c82d1913665774f29fc2caf46a28eed9e0fa7a66c6d3e WatchSource:0}: Error finding container 2cb62740cc3397d6ed2c82d1913665774f29fc2caf46a28eed9e0fa7a66c6d3e: Status 404 returned error can't find the container with id 2cb62740cc3397d6ed2c82d1913665774f29fc2caf46a28eed9e0fa7a66c6d3e Jun 02 01:44:10.366108 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:10.366019 2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-657f6fdbb8-cntl6" Jun 02 01:44:10.445656 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:10.445609 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5768fcc9dd-w9b6j" event={"ID":"1b8379d9-608a-4c5b-a377-b4b69afb78b7","Type":"ContainerStarted","Data":"2cb62740cc3397d6ed2c82d1913665774f29fc2caf46a28eed9e0fa7a66c6d3e"} Jun 02 01:44:11.451984 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:11.451948 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7f4e3de2-1d34-464b-8dc3-dd008db7de61","Type":"ContainerStarted","Data":"f2723229af9eb6f6bf1a75163ef58ae8e131067926cc3c5adc8218190a0ede9e"} Jun 02 01:44:11.451984 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:11.451990 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7f4e3de2-1d34-464b-8dc3-dd008db7de61","Type":"ContainerStarted","Data":"52491bf8a482b3bd47591f994267fce883baaf59f7e017d029c8379e9ae5f180"} Jun 02 01:44:11.452455 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:11.452001 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7f4e3de2-1d34-464b-8dc3-dd008db7de61","Type":"ContainerStarted","Data":"0ff8ace61b1774c90819d14af70fc704a459f7537050abdf904e1688d45d8f1d"} Jun 02 01:44:11.452455 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:11.452009 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7f4e3de2-1d34-464b-8dc3-dd008db7de61","Type":"ContainerStarted","Data":"45aad7f9a9f4fbf824b96fa5c8e061d8d995e2c64eb4888f0f065b4d6416bd72"} Jun 02 01:44:11.452455 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:11.452018 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7f4e3de2-1d34-464b-8dc3-dd008db7de61","Type":"ContainerStarted","Data":"6a8c80d8c59dfa05bb6c68062c79c1d8cf72cc2a24c0fb5a370ef98355c7550b"} Jun 02 01:44:12.455800 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:12.455761 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5768fcc9dd-w9b6j" event={"ID":"1b8379d9-608a-4c5b-a377-b4b69afb78b7","Type":"ContainerStarted","Data":"25efcd8d301ac9caa7e010095e44ab55aacdc63827a089df5855a25718967055"} Jun 02 01:44:12.456278 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:12.455910 2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5768fcc9dd-w9b6j" Jun 02 01:44:12.458778 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:12.458749 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7f4e3de2-1d34-464b-8dc3-dd008db7de61","Type":"ContainerStarted","Data":"d93674c92301e69300d0315e6cfd29fea1e78362f7d3a4f89d7a519678c8e5e0"} Jun 02 01:44:12.461095 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:12.461077 2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5768fcc9dd-w9b6j" Jun 02 01:44:12.469531 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:12.469458 2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5768fcc9dd-w9b6j" podStartSLOduration=1.855140316 podStartE2EDuration="3.469442791s" podCreationTimestamp="2026-06-02 01:44:09 +0000 UTC" firstStartedPulling="2026-06-02 01:44:10.111264153 +0000 UTC m=+167.757070772" lastFinishedPulling="2026-06-02 01:44:11.725566627 +0000 UTC m=+169.371373247" observedRunningTime="2026-06-02 01:44:12.468880329 +0000 UTC m=+170.114686971" watchObservedRunningTime="2026-06-02 01:44:12.469442791 +0000 UTC m=+170.115249436" Jun 02 01:44:12.509034 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:12.508985 2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.752778883 podStartE2EDuration="7.508971934s" podCreationTimestamp="2026-06-02 01:44:05 +0000 UTC" firstStartedPulling="2026-06-02 01:44:07.301125997 +0000 UTC m=+164.946932617" lastFinishedPulling="2026-06-02 01:44:12.057319049 +0000 UTC m=+169.703125668" observedRunningTime="2026-06-02 01:44:12.506320557 +0000 UTC m=+170.152127199" watchObservedRunningTime="2026-06-02 01:44:12.508971934 +0000 UTC m=+170.154778576" Jun 02 01:44:17.441788 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:17.441750 2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-fs4kq" Jun 02 01:44:32.337623 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:32.337592 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-m4mvd_e5cbb2d6-6afa-4be3-9aab-2f3bd30125bd/serve-healthcheck-canary/0.log" Jun 02 01:44:36.785080 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:36.785041 2571 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d55995968-5zw74" podUID="ecb04a0f-96b4-46f0-baa3-cac95651f7bf" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jun 02 01:44:46.784733 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:46.784689 2571 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d55995968-5zw74" podUID="ecb04a0f-96b4-46f0-baa3-cac95651f7bf" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jun 02 01:44:47.552354 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:47.552320 2571 generic.go:358] "Generic (PLEG): container finished" podID="48a4fcb7-07a8-4a1a-a4bb-ca854f1bd6c0" containerID="8f836fc4b6c57901ddeb371c372c231ce4cf2e0544b1cce22a6d079e06a28909" exitCode=0 Jun 02 01:44:47.552518 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:47.552379 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-69877444c-rzbt6" event={"ID":"48a4fcb7-07a8-4a1a-a4bb-ca854f1bd6c0","Type":"ContainerDied","Data":"8f836fc4b6c57901ddeb371c372c231ce4cf2e0544b1cce22a6d079e06a28909"} Jun 02 01:44:47.552674 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:47.552662 2571 scope.go:117] "RemoveContainer" containerID="8f836fc4b6c57901ddeb371c372c231ce4cf2e0544b1cce22a6d079e06a28909" Jun 02 01:44:48.556323 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:48.556285 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-69877444c-rzbt6" event={"ID":"48a4fcb7-07a8-4a1a-a4bb-ca854f1bd6c0","Type":"ContainerStarted","Data":"07202727dfafcc1cf2a33e9c819eadf200b5b1c13fc1d48752f681c29515b19a"} Jun 02 01:44:56.784823 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:56.784782 2571 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d55995968-5zw74" podUID="ecb04a0f-96b4-46f0-baa3-cac95651f7bf" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jun 02 01:44:56.785221 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:56.784861 2571 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d55995968-5zw74" Jun 02 01:44:56.785354 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:56.785323 2571 kuberuntime_manager.go:1107] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"e6dd72a3f5c07e4b39333baab0f4db8f949b9d606c5a63aefd5d9c95a5b2b07b"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d55995968-5zw74" containerMessage="Container service-proxy failed liveness probe, will be restarted" Jun 02 01:44:56.785403 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:56.785388 2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d55995968-5zw74" podUID="ecb04a0f-96b4-46f0-baa3-cac95651f7bf" containerName="service-proxy" containerID="cri-o://e6dd72a3f5c07e4b39333baab0f4db8f949b9d606c5a63aefd5d9c95a5b2b07b" gracePeriod=30 Jun 02 01:44:57.587106 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:57.587068 2571 generic.go:358] "Generic (PLEG): container finished" podID="ecb04a0f-96b4-46f0-baa3-cac95651f7bf" containerID="e6dd72a3f5c07e4b39333baab0f4db8f949b9d606c5a63aefd5d9c95a5b2b07b" exitCode=2 Jun 02 01:44:57.587292 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:57.587109 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d55995968-5zw74" event={"ID":"ecb04a0f-96b4-46f0-baa3-cac95651f7bf","Type":"ContainerDied","Data":"e6dd72a3f5c07e4b39333baab0f4db8f949b9d606c5a63aefd5d9c95a5b2b07b"} Jun 02 01:44:57.587292 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:44:57.587161 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d55995968-5zw74" event={"ID":"ecb04a0f-96b4-46f0-baa3-cac95651f7bf","Type":"ContainerStarted","Data":"3c569ccd5d7b9857ed6e9a743be915fd850bb79c00bc95c24cfa00f9cc6da3f2"} Jun 02 01:45:28.575742 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:45:28.575707 2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5b855cb479-9wn4b"] Jun 02 01:45:28.578476 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:45:28.578460 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5b855cb479-9wn4b" Jun 02 01:45:28.580937 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:45:28.580912 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jun 02 01:45:28.581083 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:45:28.581010 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jun 02 01:45:28.581083 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:45:28.581020 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-vplxq\"" Jun 02 01:45:28.581239 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:45:28.581095 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jun 02 01:45:28.581449 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:45:28.581433 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jun 02 01:45:28.581507 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:45:28.581468 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jun 02 01:45:28.585882 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:45:28.585859 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jun 02 01:45:28.588357 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:45:28.588334 2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5b855cb479-9wn4b"] Jun 02 01:45:28.608763 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:45:28.608737 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d7e8c237-8e5c-44ac-8ade-bdab1fabd39a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5b855cb479-9wn4b\" (UID: \"d7e8c237-8e5c-44ac-8ade-bdab1fabd39a\") " pod="openshift-monitoring/telemeter-client-5b855cb479-9wn4b" Jun 02 01:45:28.608856 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:45:28.608769 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d7e8c237-8e5c-44ac-8ade-bdab1fabd39a-telemeter-client-tls\") pod \"telemeter-client-5b855cb479-9wn4b\" (UID: \"d7e8c237-8e5c-44ac-8ade-bdab1fabd39a\") " pod="openshift-monitoring/telemeter-client-5b855cb479-9wn4b" Jun 02 01:45:28.608856 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:45:28.608805 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d7e8c237-8e5c-44ac-8ade-bdab1fabd39a-secret-telemeter-client\") pod \"telemeter-client-5b855cb479-9wn4b\" (UID: \"d7e8c237-8e5c-44ac-8ade-bdab1fabd39a\") " pod="openshift-monitoring/telemeter-client-5b855cb479-9wn4b" Jun 02 01:45:28.608856 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:45:28.608852 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7e8c237-8e5c-44ac-8ade-bdab1fabd39a-serving-certs-ca-bundle\") pod \"telemeter-client-5b855cb479-9wn4b\" (UID: \"d7e8c237-8e5c-44ac-8ade-bdab1fabd39a\") " pod="openshift-monitoring/telemeter-client-5b855cb479-9wn4b" Jun 02 01:45:28.608959 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:45:28.608873 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d7e8c237-8e5c-44ac-8ade-bdab1fabd39a-metrics-client-ca\") pod \"telemeter-client-5b855cb479-9wn4b\" (UID: \"d7e8c237-8e5c-44ac-8ade-bdab1fabd39a\") " pod="openshift-monitoring/telemeter-client-5b855cb479-9wn4b" Jun 02 01:45:28.608959 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:45:28.608889 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5zlg\" (UniqueName: \"kubernetes.io/projected/d7e8c237-8e5c-44ac-8ade-bdab1fabd39a-kube-api-access-z5zlg\") pod \"telemeter-client-5b855cb479-9wn4b\" (UID: \"d7e8c237-8e5c-44ac-8ade-bdab1fabd39a\") " pod="openshift-monitoring/telemeter-client-5b855cb479-9wn4b" Jun 02 01:45:28.608959 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:45:28.608945 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7e8c237-8e5c-44ac-8ade-bdab1fabd39a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5b855cb479-9wn4b\" (UID: \"d7e8c237-8e5c-44ac-8ade-bdab1fabd39a\") " pod="openshift-monitoring/telemeter-client-5b855cb479-9wn4b" Jun 02 01:45:28.609053 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:45:28.608973 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d7e8c237-8e5c-44ac-8ade-bdab1fabd39a-federate-client-tls\") pod \"telemeter-client-5b855cb479-9wn4b\" (UID: \"d7e8c237-8e5c-44ac-8ade-bdab1fabd39a\") " pod="openshift-monitoring/telemeter-client-5b855cb479-9wn4b" Jun 02 01:45:28.709303 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:45:28.709275 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d7e8c237-8e5c-44ac-8ade-bdab1fabd39a-secret-telemeter-client\") pod \"telemeter-client-5b855cb479-9wn4b\" (UID: \"d7e8c237-8e5c-44ac-8ade-bdab1fabd39a\") " pod="openshift-monitoring/telemeter-client-5b855cb479-9wn4b" Jun 02 01:45:28.709440 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:45:28.709311 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7e8c237-8e5c-44ac-8ade-bdab1fabd39a-serving-certs-ca-bundle\") pod \"telemeter-client-5b855cb479-9wn4b\" (UID: \"d7e8c237-8e5c-44ac-8ade-bdab1fabd39a\") " pod="openshift-monitoring/telemeter-client-5b855cb479-9wn4b" Jun 02 01:45:28.709440 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:45:28.709332 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d7e8c237-8e5c-44ac-8ade-bdab1fabd39a-metrics-client-ca\") pod \"telemeter-client-5b855cb479-9wn4b\" (UID: \"d7e8c237-8e5c-44ac-8ade-bdab1fabd39a\") " pod="openshift-monitoring/telemeter-client-5b855cb479-9wn4b" Jun 02 01:45:28.709440 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:45:28.709348 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-z5zlg\" (UniqueName: \"kubernetes.io/projected/d7e8c237-8e5c-44ac-8ade-bdab1fabd39a-kube-api-access-z5zlg\") pod \"telemeter-client-5b855cb479-9wn4b\" (UID: \"d7e8c237-8e5c-44ac-8ade-bdab1fabd39a\") " pod="openshift-monitoring/telemeter-client-5b855cb479-9wn4b" Jun 02 01:45:28.709440 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:45:28.709370 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7e8c237-8e5c-44ac-8ade-bdab1fabd39a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5b855cb479-9wn4b\" (UID: \"d7e8c237-8e5c-44ac-8ade-bdab1fabd39a\") " pod="openshift-monitoring/telemeter-client-5b855cb479-9wn4b" Jun 02 01:45:28.709440 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:45:28.709390 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d7e8c237-8e5c-44ac-8ade-bdab1fabd39a-federate-client-tls\") pod \"telemeter-client-5b855cb479-9wn4b\" (UID: \"d7e8c237-8e5c-44ac-8ade-bdab1fabd39a\") " pod="openshift-monitoring/telemeter-client-5b855cb479-9wn4b" Jun 02 01:45:28.709440 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:45:28.709410 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d7e8c237-8e5c-44ac-8ade-bdab1fabd39a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5b855cb479-9wn4b\" (UID: \"d7e8c237-8e5c-44ac-8ade-bdab1fabd39a\") " pod="openshift-monitoring/telemeter-client-5b855cb479-9wn4b" Jun 02 01:45:28.709842 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:45:28.709444 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d7e8c237-8e5c-44ac-8ade-bdab1fabd39a-telemeter-client-tls\") pod \"telemeter-client-5b855cb479-9wn4b\" (UID: \"d7e8c237-8e5c-44ac-8ade-bdab1fabd39a\") " pod="openshift-monitoring/telemeter-client-5b855cb479-9wn4b" Jun 02 01:45:28.710107 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:45:28.710084 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7e8c237-8e5c-44ac-8ade-bdab1fabd39a-serving-certs-ca-bundle\") pod \"telemeter-client-5b855cb479-9wn4b\" (UID: \"d7e8c237-8e5c-44ac-8ade-bdab1fabd39a\") " pod="openshift-monitoring/telemeter-client-5b855cb479-9wn4b" Jun 02 01:45:28.710271 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:45:28.710207 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d7e8c237-8e5c-44ac-8ade-bdab1fabd39a-metrics-client-ca\") pod \"telemeter-client-5b855cb479-9wn4b\" (UID: \"d7e8c237-8e5c-44ac-8ade-bdab1fabd39a\") " pod="openshift-monitoring/telemeter-client-5b855cb479-9wn4b" Jun 02 01:45:28.710271 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:45:28.710264 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7e8c237-8e5c-44ac-8ade-bdab1fabd39a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5b855cb479-9wn4b\" (UID: \"d7e8c237-8e5c-44ac-8ade-bdab1fabd39a\") " pod="openshift-monitoring/telemeter-client-5b855cb479-9wn4b" Jun 02 01:45:28.712057 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:45:28.712033 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d7e8c237-8e5c-44ac-8ade-bdab1fabd39a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5b855cb479-9wn4b\" (UID: \"d7e8c237-8e5c-44ac-8ade-bdab1fabd39a\") " pod="openshift-monitoring/telemeter-client-5b855cb479-9wn4b" Jun 02 01:45:28.712175 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:45:28.712158 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d7e8c237-8e5c-44ac-8ade-bdab1fabd39a-telemeter-client-tls\") pod \"telemeter-client-5b855cb479-9wn4b\" (UID: \"d7e8c237-8e5c-44ac-8ade-bdab1fabd39a\") " pod="openshift-monitoring/telemeter-client-5b855cb479-9wn4b" Jun 02 01:45:28.712300 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:45:28.712285 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d7e8c237-8e5c-44ac-8ade-bdab1fabd39a-federate-client-tls\") pod \"telemeter-client-5b855cb479-9wn4b\" (UID: \"d7e8c237-8e5c-44ac-8ade-bdab1fabd39a\") " pod="openshift-monitoring/telemeter-client-5b855cb479-9wn4b" Jun 02 01:45:28.712425 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:45:28.712405 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d7e8c237-8e5c-44ac-8ade-bdab1fabd39a-secret-telemeter-client\") pod \"telemeter-client-5b855cb479-9wn4b\" (UID: \"d7e8c237-8e5c-44ac-8ade-bdab1fabd39a\") " pod="openshift-monitoring/telemeter-client-5b855cb479-9wn4b" Jun 02 01:45:28.717656 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:45:28.717634 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5zlg\" (UniqueName: \"kubernetes.io/projected/d7e8c237-8e5c-44ac-8ade-bdab1fabd39a-kube-api-access-z5zlg\") pod \"telemeter-client-5b855cb479-9wn4b\" (UID: \"d7e8c237-8e5c-44ac-8ade-bdab1fabd39a\") " pod="openshift-monitoring/telemeter-client-5b855cb479-9wn4b" Jun 02 01:45:28.887588 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:45:28.887565 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5b855cb479-9wn4b" Jun 02 01:45:29.013020 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:45:29.012986 2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5b855cb479-9wn4b"] Jun 02 01:45:29.016183 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:45:29.016109 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7e8c237_8e5c_44ac_8ade_bdab1fabd39a.slice/crio-a32f2019f30e48bc5ea7276539ecdf646aa837c797f900392ba2337f5621a57d WatchSource:0}: Error finding container a32f2019f30e48bc5ea7276539ecdf646aa837c797f900392ba2337f5621a57d: Status 404 returned error can't find the container with id a32f2019f30e48bc5ea7276539ecdf646aa837c797f900392ba2337f5621a57d Jun 02 01:45:29.676745 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:45:29.676708 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5b855cb479-9wn4b" event={"ID":"d7e8c237-8e5c-44ac-8ade-bdab1fabd39a","Type":"ContainerStarted","Data":"a32f2019f30e48bc5ea7276539ecdf646aa837c797f900392ba2337f5621a57d"} Jun 02 01:45:31.683684 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:45:31.683646 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5b855cb479-9wn4b" event={"ID":"d7e8c237-8e5c-44ac-8ade-bdab1fabd39a","Type":"ContainerStarted","Data":"871c852399a0b37cbe20819e2acdf522b098f92ea28ea1a8cc26708e98d6400f"} Jun 02 01:45:31.683684 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:45:31.683686 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5b855cb479-9wn4b" event={"ID":"d7e8c237-8e5c-44ac-8ade-bdab1fabd39a","Type":"ContainerStarted","Data":"31c2f58116ef30e867bc29c4b2cca7180dffa3b52dff8aea50d36be67392add2"} Jun 02 01:45:31.684222 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:45:31.683695 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5b855cb479-9wn4b" event={"ID":"d7e8c237-8e5c-44ac-8ade-bdab1fabd39a","Type":"ContainerStarted","Data":"1a1f43f046e4434d95f222124f54ee11a9abf4d70cf986ce69c425abe9aed065"} Jun 02 01:45:31.705050 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:45:31.704997 2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5b855cb479-9wn4b" podStartSLOduration=1.4830567860000001 podStartE2EDuration="3.704980886s" podCreationTimestamp="2026-06-02 01:45:28 +0000 UTC" firstStartedPulling="2026-06-02 01:45:29.018109018 +0000 UTC m=+246.663915651" lastFinishedPulling="2026-06-02 01:45:31.240033116 +0000 UTC m=+248.885839751" observedRunningTime="2026-06-02 01:45:31.703058514 +0000 UTC m=+249.348865156" watchObservedRunningTime="2026-06-02 01:45:31.704980886 +0000 UTC m=+249.350787533" Jun 02 01:46:22.827745 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:46:22.827721 2571 kubelet.go:1628] "Image garbage collection succeeded" Jun 02 01:46:34.079955 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:46:34.079917 2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-lqc45"] Jun 02 01:46:34.083251 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:46:34.083234 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqc45" Jun 02 01:46:34.085948 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:46:34.085924 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jun 02 01:46:34.090768 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:46:34.090748 2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lqc45"] Jun 02 01:46:34.207142 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:46:34.207089 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7fd14a82-6157-4b43-ac1d-497b935341a5-dbus\") pod \"global-pull-secret-syncer-lqc45\" (UID: \"7fd14a82-6157-4b43-ac1d-497b935341a5\") " pod="kube-system/global-pull-secret-syncer-lqc45" Jun 02 01:46:34.207298 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:46:34.207148 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7fd14a82-6157-4b43-ac1d-497b935341a5-original-pull-secret\") pod \"global-pull-secret-syncer-lqc45\" (UID: \"7fd14a82-6157-4b43-ac1d-497b935341a5\") " pod="kube-system/global-pull-secret-syncer-lqc45" Jun 02 01:46:34.207298 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:46:34.207259 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7fd14a82-6157-4b43-ac1d-497b935341a5-kubelet-config\") pod \"global-pull-secret-syncer-lqc45\" (UID: \"7fd14a82-6157-4b43-ac1d-497b935341a5\") " pod="kube-system/global-pull-secret-syncer-lqc45" Jun 02 01:46:34.308212 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:46:34.308162 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7fd14a82-6157-4b43-ac1d-497b935341a5-dbus\") pod \"global-pull-secret-syncer-lqc45\" (UID: \"7fd14a82-6157-4b43-ac1d-497b935341a5\") " pod="kube-system/global-pull-secret-syncer-lqc45" Jun 02 01:46:34.308212 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:46:34.308211 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7fd14a82-6157-4b43-ac1d-497b935341a5-original-pull-secret\") pod \"global-pull-secret-syncer-lqc45\" (UID: \"7fd14a82-6157-4b43-ac1d-497b935341a5\") " pod="kube-system/global-pull-secret-syncer-lqc45" Jun 02 01:46:34.308424 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:46:34.308262 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7fd14a82-6157-4b43-ac1d-497b935341a5-kubelet-config\") pod \"global-pull-secret-syncer-lqc45\" (UID: \"7fd14a82-6157-4b43-ac1d-497b935341a5\") " pod="kube-system/global-pull-secret-syncer-lqc45" Jun 02 01:46:34.308424 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:46:34.308358 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7fd14a82-6157-4b43-ac1d-497b935341a5-kubelet-config\") pod \"global-pull-secret-syncer-lqc45\" (UID: \"7fd14a82-6157-4b43-ac1d-497b935341a5\") " pod="kube-system/global-pull-secret-syncer-lqc45" Jun 02 01:46:34.308424 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:46:34.308358 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7fd14a82-6157-4b43-ac1d-497b935341a5-dbus\") pod \"global-pull-secret-syncer-lqc45\" (UID: \"7fd14a82-6157-4b43-ac1d-497b935341a5\") " pod="kube-system/global-pull-secret-syncer-lqc45" Jun 02 01:46:34.310481 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:46:34.310455 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7fd14a82-6157-4b43-ac1d-497b935341a5-original-pull-secret\") pod \"global-pull-secret-syncer-lqc45\" (UID: \"7fd14a82-6157-4b43-ac1d-497b935341a5\") " pod="kube-system/global-pull-secret-syncer-lqc45" Jun 02 01:46:34.392660 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:46:34.392631 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqc45" Jun 02 01:46:34.514417 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:46:34.514303 2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lqc45"] Jun 02 01:46:34.516928 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:46:34.516905 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fd14a82_6157_4b43_ac1d_497b935341a5.slice/crio-8c7bb0fb1f5f39af8a07dc183f8b72b5c5a634dd6a3b9d318a57a338eb8dbcac WatchSource:0}: Error finding container 8c7bb0fb1f5f39af8a07dc183f8b72b5c5a634dd6a3b9d318a57a338eb8dbcac: Status 404 returned error can't find the container with id 8c7bb0fb1f5f39af8a07dc183f8b72b5c5a634dd6a3b9d318a57a338eb8dbcac Jun 02 01:46:34.518656 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:46:34.518641 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jun 02 01:46:34.856865 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:46:34.856776 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lqc45" event={"ID":"7fd14a82-6157-4b43-ac1d-497b935341a5","Type":"ContainerStarted","Data":"8c7bb0fb1f5f39af8a07dc183f8b72b5c5a634dd6a3b9d318a57a338eb8dbcac"} Jun 02 01:46:38.871029 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:46:38.870994 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lqc45" event={"ID":"7fd14a82-6157-4b43-ac1d-497b935341a5","Type":"ContainerStarted","Data":"c073dab0571d7e407b85d419207ee608cce46ee6541eb2601d6b03ddef93317c"} Jun 02 01:46:38.888060 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:46:38.887998 2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-lqc45" podStartSLOduration=0.644245469 podStartE2EDuration="4.887982298s" podCreationTimestamp="2026-06-02 01:46:34 +0000 UTC" firstStartedPulling="2026-06-02 01:46:34.518780268 +0000 UTC m=+312.164586887" lastFinishedPulling="2026-06-02 01:46:38.762517092 +0000 UTC m=+316.408323716" observedRunningTime="2026-06-02 01:46:38.887393573 +0000 UTC m=+316.533200214" watchObservedRunningTime="2026-06-02 01:46:38.887982298 +0000 UTC m=+316.533788939" Jun 02 01:47:45.927590 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:47:45.927556 2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-587ccfb98-87h28"] Jun 02 01:47:45.930439 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:47:45.930420 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-587ccfb98-87h28" Jun 02 01:47:45.933055 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:47:45.933028 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-4tct5\"" Jun 02 01:47:45.933055 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:47:45.933050 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jun 02 01:47:45.934066 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:47:45.934044 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jun 02 01:47:45.939747 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:47:45.939721 2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-587ccfb98-87h28"] Jun 02 01:47:45.968668 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:47:45.968628 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjtjj\" (UniqueName: \"kubernetes.io/projected/4648f038-975c-4485-b0bd-677ff3914e5d-kube-api-access-zjtjj\") pod \"cert-manager-webhook-587ccfb98-87h28\" (UID: \"4648f038-975c-4485-b0bd-677ff3914e5d\") " pod="cert-manager/cert-manager-webhook-587ccfb98-87h28" Jun 02 01:47:45.968842 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:47:45.968688 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4648f038-975c-4485-b0bd-677ff3914e5d-bound-sa-token\") pod \"cert-manager-webhook-587ccfb98-87h28\" (UID: \"4648f038-975c-4485-b0bd-677ff3914e5d\") " pod="cert-manager/cert-manager-webhook-587ccfb98-87h28" Jun 02 01:47:46.069780 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:47:46.069739 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zjtjj\" (UniqueName: \"kubernetes.io/projected/4648f038-975c-4485-b0bd-677ff3914e5d-kube-api-access-zjtjj\") pod \"cert-manager-webhook-587ccfb98-87h28\" (UID: \"4648f038-975c-4485-b0bd-677ff3914e5d\") " pod="cert-manager/cert-manager-webhook-587ccfb98-87h28" Jun 02 01:47:46.069950 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:47:46.069800 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4648f038-975c-4485-b0bd-677ff3914e5d-bound-sa-token\") pod \"cert-manager-webhook-587ccfb98-87h28\" (UID: \"4648f038-975c-4485-b0bd-677ff3914e5d\") " pod="cert-manager/cert-manager-webhook-587ccfb98-87h28" Jun 02 01:47:46.077517 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:47:46.077482 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4648f038-975c-4485-b0bd-677ff3914e5d-bound-sa-token\") pod \"cert-manager-webhook-587ccfb98-87h28\" (UID: \"4648f038-975c-4485-b0bd-677ff3914e5d\") " pod="cert-manager/cert-manager-webhook-587ccfb98-87h28" Jun 02 01:47:46.077647 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:47:46.077575 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjtjj\" (UniqueName: \"kubernetes.io/projected/4648f038-975c-4485-b0bd-677ff3914e5d-kube-api-access-zjtjj\") pod \"cert-manager-webhook-587ccfb98-87h28\" (UID: \"4648f038-975c-4485-b0bd-677ff3914e5d\") " pod="cert-manager/cert-manager-webhook-587ccfb98-87h28" Jun 02 01:47:46.239559 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:47:46.239467 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-587ccfb98-87h28" Jun 02 01:47:46.364853 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:47:46.364828 2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-587ccfb98-87h28"] Jun 02 01:47:46.367235 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:47:46.367207 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4648f038_975c_4485_b0bd_677ff3914e5d.slice/crio-582941a351aef0887aa7b4d3b174235d8f172f98ad1fb7e5121f2bc89c7cfbb7 WatchSource:0}: Error finding container 582941a351aef0887aa7b4d3b174235d8f172f98ad1fb7e5121f2bc89c7cfbb7: Status 404 returned error can't find the container with id 582941a351aef0887aa7b4d3b174235d8f172f98ad1fb7e5121f2bc89c7cfbb7 Jun 02 01:47:47.053322 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:47:47.053267 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-587ccfb98-87h28" event={"ID":"4648f038-975c-4485-b0bd-677ff3914e5d","Type":"ContainerStarted","Data":"582941a351aef0887aa7b4d3b174235d8f172f98ad1fb7e5121f2bc89c7cfbb7"} Jun 02 01:47:49.060928 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:47:49.060890 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-587ccfb98-87h28" event={"ID":"4648f038-975c-4485-b0bd-677ff3914e5d","Type":"ContainerStarted","Data":"8dd83b24ba289d3485d122f312dde8865b872b8fb01e7a237f48ac33449ddf2d"} Jun 02 01:47:49.061377 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:47:49.060950 2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-587ccfb98-87h28" Jun 02 01:47:49.081282 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:47:49.081221 2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-587ccfb98-87h28" podStartSLOduration=1.526590504 podStartE2EDuration="4.081207689s" podCreationTimestamp="2026-06-02 01:47:45 +0000 UTC" firstStartedPulling="2026-06-02 01:47:46.369102268 +0000 UTC m=+384.014908893" lastFinishedPulling="2026-06-02 01:47:48.923719453 +0000 UTC m=+386.569526078" observedRunningTime="2026-06-02 01:47:49.080216478 +0000 UTC m=+386.726023120" watchObservedRunningTime="2026-06-02 01:47:49.081207689 +0000 UTC m=+386.727014331" Jun 02 01:47:55.066463 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:47:55.066419 2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-587ccfb98-87h28" Jun 02 01:48:14.955297 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:14.955265 2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6fdd9d74b4-frhbs"] Jun 02 01:48:14.958511 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:14.958488 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6fdd9d74b4-frhbs" Jun 02 01:48:14.961454 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:14.961430 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jun 02 01:48:14.961583 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:14.961435 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jun 02 01:48:14.961583 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:14.961519 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-f8v9d\"" Jun 02 01:48:14.961811 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:14.961784 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jun 02 01:48:14.962762 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:14.962742 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jun 02 01:48:14.976422 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:14.976395 2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6fdd9d74b4-frhbs"] Jun 02 01:48:15.075935 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:15.075901 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ae682085-80a3-41b9-8708-95f16511d75f-webhook-cert\") pod \"opendatahub-operator-controller-manager-6fdd9d74b4-frhbs\" (UID: \"ae682085-80a3-41b9-8708-95f16511d75f\") " pod="opendatahub/opendatahub-operator-controller-manager-6fdd9d74b4-frhbs" Jun 02 01:48:15.076096 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:15.075945 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ae682085-80a3-41b9-8708-95f16511d75f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6fdd9d74b4-frhbs\" (UID: \"ae682085-80a3-41b9-8708-95f16511d75f\") " pod="opendatahub/opendatahub-operator-controller-manager-6fdd9d74b4-frhbs" Jun 02 01:48:15.076096 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:15.075993 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ae682085-80a3-41b9-8708-95f16511d75f-manifests\") pod \"opendatahub-operator-controller-manager-6fdd9d74b4-frhbs\" (UID: \"ae682085-80a3-41b9-8708-95f16511d75f\") " pod="opendatahub/opendatahub-operator-controller-manager-6fdd9d74b4-frhbs" Jun 02 01:48:15.076096 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:15.076022 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae682085-80a3-41b9-8708-95f16511d75f-tmp\") pod \"opendatahub-operator-controller-manager-6fdd9d74b4-frhbs\" (UID: \"ae682085-80a3-41b9-8708-95f16511d75f\") " pod="opendatahub/opendatahub-operator-controller-manager-6fdd9d74b4-frhbs" Jun 02 01:48:15.076096 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:15.076072 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v5j4\" (UniqueName: \"kubernetes.io/projected/ae682085-80a3-41b9-8708-95f16511d75f-kube-api-access-4v5j4\") pod \"opendatahub-operator-controller-manager-6fdd9d74b4-frhbs\" (UID: \"ae682085-80a3-41b9-8708-95f16511d75f\") " pod="opendatahub/opendatahub-operator-controller-manager-6fdd9d74b4-frhbs" Jun 02 01:48:15.177170 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:15.177103 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ae682085-80a3-41b9-8708-95f16511d75f-webhook-cert\") pod \"opendatahub-operator-controller-manager-6fdd9d74b4-frhbs\" (UID: \"ae682085-80a3-41b9-8708-95f16511d75f\") " pod="opendatahub/opendatahub-operator-controller-manager-6fdd9d74b4-frhbs" Jun 02 01:48:15.177170 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:15.177182 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ae682085-80a3-41b9-8708-95f16511d75f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6fdd9d74b4-frhbs\" (UID: \"ae682085-80a3-41b9-8708-95f16511d75f\") " pod="opendatahub/opendatahub-operator-controller-manager-6fdd9d74b4-frhbs" Jun 02 01:48:15.177396 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:15.177213 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ae682085-80a3-41b9-8708-95f16511d75f-manifests\") pod \"opendatahub-operator-controller-manager-6fdd9d74b4-frhbs\" (UID: \"ae682085-80a3-41b9-8708-95f16511d75f\") " pod="opendatahub/opendatahub-operator-controller-manager-6fdd9d74b4-frhbs" Jun 02 01:48:15.177396 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:15.177237 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae682085-80a3-41b9-8708-95f16511d75f-tmp\") pod \"opendatahub-operator-controller-manager-6fdd9d74b4-frhbs\" (UID: \"ae682085-80a3-41b9-8708-95f16511d75f\") " pod="opendatahub/opendatahub-operator-controller-manager-6fdd9d74b4-frhbs" Jun 02 01:48:15.177396 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:15.177368 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4v5j4\" (UniqueName: \"kubernetes.io/projected/ae682085-80a3-41b9-8708-95f16511d75f-kube-api-access-4v5j4\") pod \"opendatahub-operator-controller-manager-6fdd9d74b4-frhbs\" (UID: \"ae682085-80a3-41b9-8708-95f16511d75f\") " pod="opendatahub/opendatahub-operator-controller-manager-6fdd9d74b4-frhbs" Jun 02 01:48:15.177674 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:15.177642 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ae682085-80a3-41b9-8708-95f16511d75f-manifests\") pod \"opendatahub-operator-controller-manager-6fdd9d74b4-frhbs\" (UID: \"ae682085-80a3-41b9-8708-95f16511d75f\") " pod="opendatahub/opendatahub-operator-controller-manager-6fdd9d74b4-frhbs" Jun 02 01:48:15.179428 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:15.179400 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae682085-80a3-41b9-8708-95f16511d75f-tmp\") pod \"opendatahub-operator-controller-manager-6fdd9d74b4-frhbs\" (UID: \"ae682085-80a3-41b9-8708-95f16511d75f\") " pod="opendatahub/opendatahub-operator-controller-manager-6fdd9d74b4-frhbs" Jun 02 01:48:15.179540 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:15.179523 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ae682085-80a3-41b9-8708-95f16511d75f-webhook-cert\") pod \"opendatahub-operator-controller-manager-6fdd9d74b4-frhbs\" (UID: \"ae682085-80a3-41b9-8708-95f16511d75f\") " pod="opendatahub/opendatahub-operator-controller-manager-6fdd9d74b4-frhbs" Jun 02 01:48:15.179617 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:15.179600 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ae682085-80a3-41b9-8708-95f16511d75f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6fdd9d74b4-frhbs\" (UID: \"ae682085-80a3-41b9-8708-95f16511d75f\") " pod="opendatahub/opendatahub-operator-controller-manager-6fdd9d74b4-frhbs" Jun 02 01:48:15.186245 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:15.186221 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v5j4\" (UniqueName: \"kubernetes.io/projected/ae682085-80a3-41b9-8708-95f16511d75f-kube-api-access-4v5j4\") pod \"opendatahub-operator-controller-manager-6fdd9d74b4-frhbs\" (UID: \"ae682085-80a3-41b9-8708-95f16511d75f\") " pod="opendatahub/opendatahub-operator-controller-manager-6fdd9d74b4-frhbs" Jun 02 01:48:15.269297 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:15.269211 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6fdd9d74b4-frhbs" Jun 02 01:48:15.391581 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:15.391553 2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6fdd9d74b4-frhbs"] Jun 02 01:48:15.394243 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:48:15.394214 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae682085_80a3_41b9_8708_95f16511d75f.slice/crio-72c0a29e53143b81eacbefab1204d97c8a87b7d9d23c7f7be386f98c12a673ea WatchSource:0}: Error finding container 72c0a29e53143b81eacbefab1204d97c8a87b7d9d23c7f7be386f98c12a673ea: Status 404 returned error can't find the container with id 72c0a29e53143b81eacbefab1204d97c8a87b7d9d23c7f7be386f98c12a673ea Jun 02 01:48:16.139281 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:16.139238 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6fdd9d74b4-frhbs" event={"ID":"ae682085-80a3-41b9-8708-95f16511d75f","Type":"ContainerStarted","Data":"72c0a29e53143b81eacbefab1204d97c8a87b7d9d23c7f7be386f98c12a673ea"} Jun 02 01:48:19.149496 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:19.149468 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6fdd9d74b4-frhbs" event={"ID":"ae682085-80a3-41b9-8708-95f16511d75f","Type":"ContainerStarted","Data":"3b2438f9dc1a1d67c0f2ecc4d065c54ce51696f3e062303b0d74694e00b87f3f"} Jun 02 01:48:20.154591 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:20.154557 2571 generic.go:358] "Generic (PLEG): container finished" podID="ae682085-80a3-41b9-8708-95f16511d75f" containerID="3b2438f9dc1a1d67c0f2ecc4d065c54ce51696f3e062303b0d74694e00b87f3f" exitCode=0 Jun 02 01:48:20.155060 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:20.154640 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6fdd9d74b4-frhbs" event={"ID":"ae682085-80a3-41b9-8708-95f16511d75f","Type":"ContainerDied","Data":"3b2438f9dc1a1d67c0f2ecc4d065c54ce51696f3e062303b0d74694e00b87f3f"} Jun 02 01:48:20.155060 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:20.154680 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6fdd9d74b4-frhbs" event={"ID":"ae682085-80a3-41b9-8708-95f16511d75f","Type":"ContainerStarted","Data":"2d3af6db334ac735da583ddf3fc32d68a2d3996338574ea0dc0bbc163a218d66"} Jun 02 01:48:20.159150 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:20.155513 2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6fdd9d74b4-frhbs" Jun 02 01:48:20.179054 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:20.178994 2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6fdd9d74b4-frhbs" podStartSLOduration=2.01126023 podStartE2EDuration="6.178973426s" podCreationTimestamp="2026-06-02 01:48:14 +0000 UTC" firstStartedPulling="2026-06-02 01:48:15.395861219 +0000 UTC m=+413.041667839" lastFinishedPulling="2026-06-02 01:48:19.563574413 +0000 UTC m=+417.209381035" observedRunningTime="2026-06-02 01:48:20.176353295 +0000 UTC m=+417.822159936" watchObservedRunningTime="2026-06-02 01:48:20.178973426 +0000 UTC m=+417.824780069" Jun 02 01:48:31.159464 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:31.159431 2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6fdd9d74b4-frhbs" Jun 02 01:48:33.719852 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:33.719822 2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5f795fc994-hjljs"] Jun 02 01:48:33.729047 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:33.729018 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5f795fc994-hjljs" Jun 02 01:48:33.735705 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:33.735422 2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5f795fc994-hjljs"] Jun 02 01:48:33.735705 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:33.735502 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jun 02 01:48:33.735705 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:33.735513 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-ff4ws\"" Jun 02 01:48:33.735705 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:33.735529 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jun 02 01:48:33.735983 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:33.735836 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jun 02 01:48:33.736672 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:33.736247 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jun 02 01:48:33.827241 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:33.827171 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/396a0364-b4ad-49eb-8c61-0ed962f7d4d7-tmp\") pod \"kube-auth-proxy-5f795fc994-hjljs\" (UID: \"396a0364-b4ad-49eb-8c61-0ed962f7d4d7\") " pod="openshift-ingress/kube-auth-proxy-5f795fc994-hjljs" Jun 02 01:48:33.827405 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:33.827300 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwvgn\" (UniqueName: \"kubernetes.io/projected/396a0364-b4ad-49eb-8c61-0ed962f7d4d7-kube-api-access-hwvgn\") pod \"kube-auth-proxy-5f795fc994-hjljs\" (UID: \"396a0364-b4ad-49eb-8c61-0ed962f7d4d7\") " pod="openshift-ingress/kube-auth-proxy-5f795fc994-hjljs" Jun 02 01:48:33.827405 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:33.827388 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/396a0364-b4ad-49eb-8c61-0ed962f7d4d7-tls-certs\") pod \"kube-auth-proxy-5f795fc994-hjljs\" (UID: \"396a0364-b4ad-49eb-8c61-0ed962f7d4d7\") " pod="openshift-ingress/kube-auth-proxy-5f795fc994-hjljs" Jun 02 01:48:33.848352 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:33.848321 2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-76588b95-jpnzx"] Jun 02 01:48:33.851877 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:33.851859 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-76588b95-jpnzx" Jun 02 01:48:33.854450 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:33.854432 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jun 02 01:48:33.854762 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:33.854743 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-4tnj7\"" Jun 02 01:48:33.861353 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:33.861333 2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-76588b95-jpnzx"] Jun 02 01:48:33.928192 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:33.928157 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4czz\" (UniqueName: \"kubernetes.io/projected/46600c1c-d993-4e0c-9304-e4f361bed002-kube-api-access-l4czz\") pod \"odh-model-controller-76588b95-jpnzx\" (UID: \"46600c1c-d993-4e0c-9304-e4f361bed002\") " pod="opendatahub/odh-model-controller-76588b95-jpnzx" Jun 02 01:48:33.928334 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:33.928247 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/396a0364-b4ad-49eb-8c61-0ed962f7d4d7-tls-certs\") pod \"kube-auth-proxy-5f795fc994-hjljs\" (UID: \"396a0364-b4ad-49eb-8c61-0ed962f7d4d7\") " pod="openshift-ingress/kube-auth-proxy-5f795fc994-hjljs" Jun 02 01:48:33.928334 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:33.928298 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/396a0364-b4ad-49eb-8c61-0ed962f7d4d7-tmp\") pod \"kube-auth-proxy-5f795fc994-hjljs\" (UID: \"396a0364-b4ad-49eb-8c61-0ed962f7d4d7\") " pod="openshift-ingress/kube-auth-proxy-5f795fc994-hjljs" Jun 02 01:48:33.928410 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:33.928361 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hwvgn\" (UniqueName: \"kubernetes.io/projected/396a0364-b4ad-49eb-8c61-0ed962f7d4d7-kube-api-access-hwvgn\") pod \"kube-auth-proxy-5f795fc994-hjljs\" (UID: \"396a0364-b4ad-49eb-8c61-0ed962f7d4d7\") " pod="openshift-ingress/kube-auth-proxy-5f795fc994-hjljs" Jun 02 01:48:33.928410 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:33.928391 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/46600c1c-d993-4e0c-9304-e4f361bed002-cert\") pod \"odh-model-controller-76588b95-jpnzx\" (UID: \"46600c1c-d993-4e0c-9304-e4f361bed002\") " pod="opendatahub/odh-model-controller-76588b95-jpnzx" Jun 02 01:48:33.930569 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:33.930537 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/396a0364-b4ad-49eb-8c61-0ed962f7d4d7-tmp\") pod \"kube-auth-proxy-5f795fc994-hjljs\" (UID: \"396a0364-b4ad-49eb-8c61-0ed962f7d4d7\") " pod="openshift-ingress/kube-auth-proxy-5f795fc994-hjljs" Jun 02 01:48:33.930873 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:33.930851 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/396a0364-b4ad-49eb-8c61-0ed962f7d4d7-tls-certs\") pod \"kube-auth-proxy-5f795fc994-hjljs\" (UID: \"396a0364-b4ad-49eb-8c61-0ed962f7d4d7\") " pod="openshift-ingress/kube-auth-proxy-5f795fc994-hjljs" Jun 02 01:48:33.942613 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:33.942588 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwvgn\" (UniqueName: \"kubernetes.io/projected/396a0364-b4ad-49eb-8c61-0ed962f7d4d7-kube-api-access-hwvgn\") pod \"kube-auth-proxy-5f795fc994-hjljs\" (UID: \"396a0364-b4ad-49eb-8c61-0ed962f7d4d7\") " pod="openshift-ingress/kube-auth-proxy-5f795fc994-hjljs" Jun 02 01:48:34.029591 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:34.029504 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/46600c1c-d993-4e0c-9304-e4f361bed002-cert\") pod \"odh-model-controller-76588b95-jpnzx\" (UID: \"46600c1c-d993-4e0c-9304-e4f361bed002\") " pod="opendatahub/odh-model-controller-76588b95-jpnzx" Jun 02 01:48:34.029591 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:34.029559 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-l4czz\" (UniqueName: \"kubernetes.io/projected/46600c1c-d993-4e0c-9304-e4f361bed002-kube-api-access-l4czz\") pod \"odh-model-controller-76588b95-jpnzx\" (UID: \"46600c1c-d993-4e0c-9304-e4f361bed002\") " pod="opendatahub/odh-model-controller-76588b95-jpnzx" Jun 02 01:48:34.029756 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:48:34.029638 2571 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jun 02 01:48:34.029756 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:48:34.029705 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46600c1c-d993-4e0c-9304-e4f361bed002-cert podName:46600c1c-d993-4e0c-9304-e4f361bed002 nodeName:}" failed. No retries permitted until 2026-06-02 01:48:34.52968843 +0000 UTC m=+432.175495050 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/46600c1c-d993-4e0c-9304-e4f361bed002-cert") pod "odh-model-controller-76588b95-jpnzx" (UID: "46600c1c-d993-4e0c-9304-e4f361bed002") : secret "odh-model-controller-webhook-cert" not found Jun 02 01:48:34.044302 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:34.044272 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5f795fc994-hjljs" Jun 02 01:48:34.047021 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:34.046995 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4czz\" (UniqueName: \"kubernetes.io/projected/46600c1c-d993-4e0c-9304-e4f361bed002-kube-api-access-l4czz\") pod \"odh-model-controller-76588b95-jpnzx\" (UID: \"46600c1c-d993-4e0c-9304-e4f361bed002\") " pod="opendatahub/odh-model-controller-76588b95-jpnzx" Jun 02 01:48:34.187162 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:34.187090 2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5f795fc994-hjljs"] Jun 02 01:48:34.190443 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:48:34.190414 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod396a0364_b4ad_49eb_8c61_0ed962f7d4d7.slice/crio-06fe564a14f93c9c129f3b985067cb749adc842a5c5791003338e0e613f5ea5f WatchSource:0}: Error finding container 06fe564a14f93c9c129f3b985067cb749adc842a5c5791003338e0e613f5ea5f: Status 404 returned error can't find the container with id 06fe564a14f93c9c129f3b985067cb749adc842a5c5791003338e0e613f5ea5f Jun 02 01:48:34.533673 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:34.533633 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/46600c1c-d993-4e0c-9304-e4f361bed002-cert\") pod \"odh-model-controller-76588b95-jpnzx\" (UID: \"46600c1c-d993-4e0c-9304-e4f361bed002\") " pod="opendatahub/odh-model-controller-76588b95-jpnzx" Jun 02 01:48:34.536353 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:34.536324 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/46600c1c-d993-4e0c-9304-e4f361bed002-cert\") pod \"odh-model-controller-76588b95-jpnzx\" (UID: \"46600c1c-d993-4e0c-9304-e4f361bed002\") " pod="opendatahub/odh-model-controller-76588b95-jpnzx" Jun 02 01:48:34.763831 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:34.763327 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-76588b95-jpnzx" Jun 02 01:48:34.931979 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:34.931759 2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-76588b95-jpnzx"] Jun 02 01:48:34.938211 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:48:34.938176 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46600c1c_d993_4e0c_9304_e4f361bed002.slice/crio-b6a128adf3b1d28c7865db33d9270a7e78500cc2e56c3bfde282fbe3d19c24f1 WatchSource:0}: Error finding container b6a128adf3b1d28c7865db33d9270a7e78500cc2e56c3bfde282fbe3d19c24f1: Status 404 returned error can't find the container with id b6a128adf3b1d28c7865db33d9270a7e78500cc2e56c3bfde282fbe3d19c24f1 Jun 02 01:48:35.200225 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:35.200181 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-76588b95-jpnzx" event={"ID":"46600c1c-d993-4e0c-9304-e4f361bed002","Type":"ContainerStarted","Data":"b6a128adf3b1d28c7865db33d9270a7e78500cc2e56c3bfde282fbe3d19c24f1"} Jun 02 01:48:35.202077 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:35.202023 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5f795fc994-hjljs" event={"ID":"396a0364-b4ad-49eb-8c61-0ed962f7d4d7","Type":"ContainerStarted","Data":"06fe564a14f93c9c129f3b985067cb749adc842a5c5791003338e0e613f5ea5f"} Jun 02 01:48:39.218187 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:39.218125 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-76588b95-jpnzx" event={"ID":"46600c1c-d993-4e0c-9304-e4f361bed002","Type":"ContainerStarted","Data":"514de3e7bbe5331cc52cb84ec4f89545dce5377995e56457e79b069c20ad572a"} Jun 02 01:48:39.218594 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:39.218361 2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-76588b95-jpnzx" Jun 02 01:48:39.219513 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:39.219479 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5f795fc994-hjljs" event={"ID":"396a0364-b4ad-49eb-8c61-0ed962f7d4d7","Type":"ContainerStarted","Data":"745ae20b73b094710fbe879a80f53b48aea2adaf4ff02da9b69e1cfdd63b16d6"} Jun 02 01:48:39.235874 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:39.235829 2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="opendatahub/odh-model-controller-76588b95-jpnzx" podStartSLOduration=2.6529910169999997 podStartE2EDuration="6.235816638s" podCreationTimestamp="2026-06-02 01:48:33 +0000 UTC" firstStartedPulling="2026-06-02 01:48:34.941580561 +0000 UTC m=+432.587387199" lastFinishedPulling="2026-06-02 01:48:38.524406188 +0000 UTC m=+436.170212820" observedRunningTime="2026-06-02 01:48:39.233599186 +0000 UTC m=+436.879405842" watchObservedRunningTime="2026-06-02 01:48:39.235816638 +0000 UTC m=+436.881623279" Jun 02 01:48:39.251212 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:39.251166 2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5f795fc994-hjljs" podStartSLOduration=1.958561935 podStartE2EDuration="6.251154134s" podCreationTimestamp="2026-06-02 01:48:33 +0000 UTC" firstStartedPulling="2026-06-02 01:48:34.192215098 +0000 UTC m=+431.838021721" lastFinishedPulling="2026-06-02 01:48:38.484807293 +0000 UTC m=+436.130613920" observedRunningTime="2026-06-02 01:48:39.249753448 +0000 UTC m=+436.895560090" watchObservedRunningTime="2026-06-02 01:48:39.251154134 +0000 UTC m=+436.896960775" Jun 02 01:48:42.932120 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:42.932084 2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-74db8bbd6d-9dx5c"] Jun 02 01:48:42.935968 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:42.935948 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-74db8bbd6d-9dx5c" Jun 02 01:48:42.936161 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:42.936032 2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-74db8bbd6d-9dx5c"] Jun 02 01:48:42.938974 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:42.938954 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jun 02 01:48:42.940167 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:42.940149 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jun 02 01:48:42.940346 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:42.940155 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-fqqtz\"" Jun 02 01:48:42.940467 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:42.940234 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jun 02 01:48:42.940467 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:42.940238 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jun 02 01:48:42.940746 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:42.940722 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jun 02 01:48:43.119770 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:43.119730 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/85e911a4-7429-4fce-9e76-b339a2a78c38-metrics-cert\") pod \"lws-controller-manager-74db8bbd6d-9dx5c\" (UID: \"85e911a4-7429-4fce-9e76-b339a2a78c38\") " pod="openshift-lws-operator/lws-controller-manager-74db8bbd6d-9dx5c" Jun 02 01:48:43.119770 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:43.119768 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85e911a4-7429-4fce-9e76-b339a2a78c38-cert\") pod \"lws-controller-manager-74db8bbd6d-9dx5c\" (UID: \"85e911a4-7429-4fce-9e76-b339a2a78c38\") " pod="openshift-lws-operator/lws-controller-manager-74db8bbd6d-9dx5c" Jun 02 01:48:43.119983 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:43.119788 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/85e911a4-7429-4fce-9e76-b339a2a78c38-manager-config\") pod \"lws-controller-manager-74db8bbd6d-9dx5c\" (UID: \"85e911a4-7429-4fce-9e76-b339a2a78c38\") " pod="openshift-lws-operator/lws-controller-manager-74db8bbd6d-9dx5c" Jun 02 01:48:43.119983 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:43.119923 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44fpw\" (UniqueName: \"kubernetes.io/projected/85e911a4-7429-4fce-9e76-b339a2a78c38-kube-api-access-44fpw\") pod \"lws-controller-manager-74db8bbd6d-9dx5c\" (UID: \"85e911a4-7429-4fce-9e76-b339a2a78c38\") " pod="openshift-lws-operator/lws-controller-manager-74db8bbd6d-9dx5c" Jun 02 01:48:43.220455 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:43.220376 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-44fpw\" (UniqueName: \"kubernetes.io/projected/85e911a4-7429-4fce-9e76-b339a2a78c38-kube-api-access-44fpw\") pod \"lws-controller-manager-74db8bbd6d-9dx5c\" (UID: \"85e911a4-7429-4fce-9e76-b339a2a78c38\") " pod="openshift-lws-operator/lws-controller-manager-74db8bbd6d-9dx5c" Jun 02 01:48:43.220455 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:43.220421 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/85e911a4-7429-4fce-9e76-b339a2a78c38-metrics-cert\") pod \"lws-controller-manager-74db8bbd6d-9dx5c\" (UID: \"85e911a4-7429-4fce-9e76-b339a2a78c38\") " pod="openshift-lws-operator/lws-controller-manager-74db8bbd6d-9dx5c" Jun 02 01:48:43.220455 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:43.220441 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85e911a4-7429-4fce-9e76-b339a2a78c38-cert\") pod \"lws-controller-manager-74db8bbd6d-9dx5c\" (UID: \"85e911a4-7429-4fce-9e76-b339a2a78c38\") " pod="openshift-lws-operator/lws-controller-manager-74db8bbd6d-9dx5c" Jun 02 01:48:43.220690 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:43.220608 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/85e911a4-7429-4fce-9e76-b339a2a78c38-manager-config\") pod \"lws-controller-manager-74db8bbd6d-9dx5c\" (UID: \"85e911a4-7429-4fce-9e76-b339a2a78c38\") " pod="openshift-lws-operator/lws-controller-manager-74db8bbd6d-9dx5c" Jun 02 01:48:43.221230 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:43.221209 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/85e911a4-7429-4fce-9e76-b339a2a78c38-manager-config\") pod \"lws-controller-manager-74db8bbd6d-9dx5c\" (UID: \"85e911a4-7429-4fce-9e76-b339a2a78c38\") " pod="openshift-lws-operator/lws-controller-manager-74db8bbd6d-9dx5c" Jun 02 01:48:43.222825 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:43.222796 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/85e911a4-7429-4fce-9e76-b339a2a78c38-metrics-cert\") pod \"lws-controller-manager-74db8bbd6d-9dx5c\" (UID: \"85e911a4-7429-4fce-9e76-b339a2a78c38\") " pod="openshift-lws-operator/lws-controller-manager-74db8bbd6d-9dx5c" Jun 02 01:48:43.222935 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:43.222907 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85e911a4-7429-4fce-9e76-b339a2a78c38-cert\") pod \"lws-controller-manager-74db8bbd6d-9dx5c\" (UID: \"85e911a4-7429-4fce-9e76-b339a2a78c38\") " pod="openshift-lws-operator/lws-controller-manager-74db8bbd6d-9dx5c" Jun 02 01:48:43.227907 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:43.227885 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44fpw\" (UniqueName: \"kubernetes.io/projected/85e911a4-7429-4fce-9e76-b339a2a78c38-kube-api-access-44fpw\") pod \"lws-controller-manager-74db8bbd6d-9dx5c\" (UID: \"85e911a4-7429-4fce-9e76-b339a2a78c38\") " pod="openshift-lws-operator/lws-controller-manager-74db8bbd6d-9dx5c" Jun 02 01:48:43.248093 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:43.248061 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-74db8bbd6d-9dx5c" Jun 02 01:48:43.372145 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:43.372099 2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-74db8bbd6d-9dx5c"] Jun 02 01:48:43.375070 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:48:43.375038 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85e911a4_7429_4fce_9e76_b339a2a78c38.slice/crio-8679b2829116707d00ee0cc9d911f550a14d66ef98b3ffaea501afe6015e4b2d WatchSource:0}: Error finding container 8679b2829116707d00ee0cc9d911f550a14d66ef98b3ffaea501afe6015e4b2d: Status 404 returned error can't find the container with id 8679b2829116707d00ee0cc9d911f550a14d66ef98b3ffaea501afe6015e4b2d Jun 02 01:48:44.238201 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:44.238161 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-74db8bbd6d-9dx5c" event={"ID":"85e911a4-7429-4fce-9e76-b339a2a78c38","Type":"ContainerStarted","Data":"8679b2829116707d00ee0cc9d911f550a14d66ef98b3ffaea501afe6015e4b2d"} Jun 02 01:48:46.246934 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:46.246895 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-74db8bbd6d-9dx5c" event={"ID":"85e911a4-7429-4fce-9e76-b339a2a78c38","Type":"ContainerStarted","Data":"7b0b53842fd1562fff68c8fdced9de19ec4be06c050ae313377495c9f1d31a75"} Jun 02 01:48:46.247386 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:46.246955 2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-74db8bbd6d-9dx5c" Jun 02 01:48:50.225203 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:50.225169 2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-76588b95-jpnzx" Jun 02 01:48:50.243537 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:50.243476 2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-74db8bbd6d-9dx5c" podStartSLOduration=5.6175322770000005 podStartE2EDuration="8.243457502s" podCreationTimestamp="2026-06-02 01:48:42 +0000 UTC" firstStartedPulling="2026-06-02 01:48:43.377017581 +0000 UTC m=+441.022824202" lastFinishedPulling="2026-06-02 01:48:46.002942803 +0000 UTC m=+443.648749427" observedRunningTime="2026-06-02 01:48:46.288222266 +0000 UTC m=+443.934029032" watchObservedRunningTime="2026-06-02 01:48:50.243457502 +0000 UTC m=+447.889264147" Jun 02 01:48:57.251847 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:48:57.251817 2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-74db8bbd6d-9dx5c" Jun 02 01:50:17.842012 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:50:17.841926 2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-6cb54b5c86-wnnv2"] Jun 02 01:50:17.845515 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:50:17.845496 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-6cb54b5c86-wnnv2" Jun 02 01:50:17.848190 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:50:17.848167 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jun 02 01:50:17.848307 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:50:17.848168 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jun 02 01:50:17.848307 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:50:17.848168 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jun 02 01:50:17.849246 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:50:17.849231 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-p7wkp\"" Jun 02 01:50:17.849299 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:50:17.849242 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jun 02 01:50:17.854227 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:50:17.854206 2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-6cb54b5c86-wnnv2"] Jun 02 01:50:17.900524 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:50:17.900486 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/34e43a88-2140-4348-901e-ffe294da6535-plugin-serving-cert\") pod \"kuadrant-console-plugin-6cb54b5c86-wnnv2\" (UID: \"34e43a88-2140-4348-901e-ffe294da6535\") " pod="kuadrant-system/kuadrant-console-plugin-6cb54b5c86-wnnv2" Jun 02 01:50:17.900715 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:50:17.900536 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/34e43a88-2140-4348-901e-ffe294da6535-nginx-conf\") pod \"kuadrant-console-plugin-6cb54b5c86-wnnv2\" (UID: \"34e43a88-2140-4348-901e-ffe294da6535\") " pod="kuadrant-system/kuadrant-console-plugin-6cb54b5c86-wnnv2" Jun 02 01:50:17.900715 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:50:17.900602 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-557cr\" (UniqueName: \"kubernetes.io/projected/34e43a88-2140-4348-901e-ffe294da6535-kube-api-access-557cr\") pod \"kuadrant-console-plugin-6cb54b5c86-wnnv2\" (UID: \"34e43a88-2140-4348-901e-ffe294da6535\") " pod="kuadrant-system/kuadrant-console-plugin-6cb54b5c86-wnnv2" Jun 02 01:50:18.001776 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:50:18.001745 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/34e43a88-2140-4348-901e-ffe294da6535-plugin-serving-cert\") pod \"kuadrant-console-plugin-6cb54b5c86-wnnv2\" (UID: \"34e43a88-2140-4348-901e-ffe294da6535\") " pod="kuadrant-system/kuadrant-console-plugin-6cb54b5c86-wnnv2" Jun 02 01:50:18.001776 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:50:18.001785 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/34e43a88-2140-4348-901e-ffe294da6535-nginx-conf\") pod \"kuadrant-console-plugin-6cb54b5c86-wnnv2\" (UID: \"34e43a88-2140-4348-901e-ffe294da6535\") " pod="kuadrant-system/kuadrant-console-plugin-6cb54b5c86-wnnv2" Jun 02 01:50:18.001987 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:50:18.001812 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-557cr\" (UniqueName: \"kubernetes.io/projected/34e43a88-2140-4348-901e-ffe294da6535-kube-api-access-557cr\") pod \"kuadrant-console-plugin-6cb54b5c86-wnnv2\" (UID: \"34e43a88-2140-4348-901e-ffe294da6535\") " pod="kuadrant-system/kuadrant-console-plugin-6cb54b5c86-wnnv2" Jun 02 01:50:18.002410 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:50:18.002386 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/34e43a88-2140-4348-901e-ffe294da6535-nginx-conf\") pod \"kuadrant-console-plugin-6cb54b5c86-wnnv2\" (UID: \"34e43a88-2140-4348-901e-ffe294da6535\") " pod="kuadrant-system/kuadrant-console-plugin-6cb54b5c86-wnnv2" Jun 02 01:50:18.004321 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:50:18.004298 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/34e43a88-2140-4348-901e-ffe294da6535-plugin-serving-cert\") pod \"kuadrant-console-plugin-6cb54b5c86-wnnv2\" (UID: \"34e43a88-2140-4348-901e-ffe294da6535\") " pod="kuadrant-system/kuadrant-console-plugin-6cb54b5c86-wnnv2" Jun 02 01:50:18.009270 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:50:18.009243 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-557cr\" (UniqueName: \"kubernetes.io/projected/34e43a88-2140-4348-901e-ffe294da6535-kube-api-access-557cr\") pod \"kuadrant-console-plugin-6cb54b5c86-wnnv2\" (UID: \"34e43a88-2140-4348-901e-ffe294da6535\") " pod="kuadrant-system/kuadrant-console-plugin-6cb54b5c86-wnnv2" Jun 02 01:50:18.156391 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:50:18.156358 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-6cb54b5c86-wnnv2" Jun 02 01:50:18.280417 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:50:18.280381 2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-6cb54b5c86-wnnv2"] Jun 02 01:50:18.283242 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:50:18.283215 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod34e43a88_2140_4348_901e_ffe294da6535.slice/crio-f642627222559a0b43b9a4125cf3d8290c7028946ca906b76f035a6cddb3eee5 WatchSource:0}: Error finding container f642627222559a0b43b9a4125cf3d8290c7028946ca906b76f035a6cddb3eee5: Status 404 returned error can't find the container with id f642627222559a0b43b9a4125cf3d8290c7028946ca906b76f035a6cddb3eee5 Jun 02 01:50:18.531567 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:50:18.531478 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-6cb54b5c86-wnnv2" event={"ID":"34e43a88-2140-4348-901e-ffe294da6535","Type":"ContainerStarted","Data":"f642627222559a0b43b9a4125cf3d8290c7028946ca906b76f035a6cddb3eee5"} Jun 02 01:50:42.618582 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:50:42.618537 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-6cb54b5c86-wnnv2" event={"ID":"34e43a88-2140-4348-901e-ffe294da6535","Type":"ContainerStarted","Data":"122cdef93c3e9af5f6a0ca369ff2f9127871f44ebf73318f6442449306c451c8"} Jun 02 01:50:42.634985 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:50:42.634935 2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-6cb54b5c86-wnnv2" podStartSLOduration=2.381102727 podStartE2EDuration="25.634921648s" podCreationTimestamp="2026-06-02 01:50:17 +0000 UTC" firstStartedPulling="2026-06-02 01:50:18.284498187 +0000 UTC m=+535.930304806" lastFinishedPulling="2026-06-02 01:50:41.538317095 +0000 UTC m=+559.184123727" observedRunningTime="2026-06-02 01:50:42.633310182 +0000 UTC m=+560.279116820" watchObservedRunningTime="2026-06-02 01:50:42.634921648 +0000 UTC m=+560.280728290" Jun 02 01:51:03.584806 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:51:03.584766 2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-f99f4b5cd-2pc9n"] Jun 02 01:51:03.622387 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:51:03.622339 2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-f99f4b5cd-2pc9n"] Jun 02 01:51:03.622547 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:51:03.622463 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f99f4b5cd-2pc9n" Jun 02 01:51:03.625055 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:51:03.625032 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-vvjqq\"" Jun 02 01:51:03.696346 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:51:03.696306 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9qpc\" (UniqueName: \"kubernetes.io/projected/ba75ed3b-9f9f-4e81-9665-5310692a5e81-kube-api-access-p9qpc\") pod \"authorino-f99f4b5cd-2pc9n\" (UID: \"ba75ed3b-9f9f-4e81-9665-5310692a5e81\") " pod="kuadrant-system/authorino-f99f4b5cd-2pc9n" Jun 02 01:51:03.759779 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:51:03.759742 2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7498df8756-zxx28"] Jun 02 01:51:03.769700 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:51:03.769673 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7498df8756-zxx28" Jun 02 01:51:03.769838 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:51:03.769806 2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7498df8756-zxx28"] Jun 02 01:51:03.797401 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:51:03.797362 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-p9qpc\" (UniqueName: \"kubernetes.io/projected/ba75ed3b-9f9f-4e81-9665-5310692a5e81-kube-api-access-p9qpc\") pod \"authorino-f99f4b5cd-2pc9n\" (UID: \"ba75ed3b-9f9f-4e81-9665-5310692a5e81\") " pod="kuadrant-system/authorino-f99f4b5cd-2pc9n" Jun 02 01:51:03.805517 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:51:03.805484 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9qpc\" (UniqueName: \"kubernetes.io/projected/ba75ed3b-9f9f-4e81-9665-5310692a5e81-kube-api-access-p9qpc\") pod \"authorino-f99f4b5cd-2pc9n\" (UID: \"ba75ed3b-9f9f-4e81-9665-5310692a5e81\") " pod="kuadrant-system/authorino-f99f4b5cd-2pc9n" Jun 02 01:51:03.898460 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:51:03.898423 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v7mh\" (UniqueName: \"kubernetes.io/projected/90d23e1c-ca7d-417c-9c8a-136ea2961e30-kube-api-access-8v7mh\") pod \"authorino-7498df8756-zxx28\" (UID: \"90d23e1c-ca7d-417c-9c8a-136ea2961e30\") " pod="kuadrant-system/authorino-7498df8756-zxx28" Jun 02 01:51:03.931469 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:51:03.931437 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f99f4b5cd-2pc9n" Jun 02 01:51:03.999893 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:51:03.999861 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8v7mh\" (UniqueName: \"kubernetes.io/projected/90d23e1c-ca7d-417c-9c8a-136ea2961e30-kube-api-access-8v7mh\") pod \"authorino-7498df8756-zxx28\" (UID: \"90d23e1c-ca7d-417c-9c8a-136ea2961e30\") " pod="kuadrant-system/authorino-7498df8756-zxx28" Jun 02 01:51:04.008825 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:51:04.008766 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v7mh\" (UniqueName: \"kubernetes.io/projected/90d23e1c-ca7d-417c-9c8a-136ea2961e30-kube-api-access-8v7mh\") pod \"authorino-7498df8756-zxx28\" (UID: \"90d23e1c-ca7d-417c-9c8a-136ea2961e30\") " pod="kuadrant-system/authorino-7498df8756-zxx28" Jun 02 01:51:04.054468 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:51:04.054439 2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-f99f4b5cd-2pc9n"] Jun 02 01:51:04.056726 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:51:04.056698 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba75ed3b_9f9f_4e81_9665_5310692a5e81.slice/crio-fe2bdb9eba841b381992a8989ff6d1ebb6eb61b5a710328e908101db7a383039 WatchSource:0}: Error finding container fe2bdb9eba841b381992a8989ff6d1ebb6eb61b5a710328e908101db7a383039: Status 404 returned error can't find the container with id fe2bdb9eba841b381992a8989ff6d1ebb6eb61b5a710328e908101db7a383039 Jun 02 01:51:04.079169 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:51:04.079124 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7498df8756-zxx28" Jun 02 01:51:04.201737 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:51:04.201706 2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7498df8756-zxx28"] Jun 02 01:51:04.205001 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:51:04.204977 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod90d23e1c_ca7d_417c_9c8a_136ea2961e30.slice/crio-c009b07d294fbd57de73d2507bbd5376490d331b320f5835a210585eb0b96797 WatchSource:0}: Error finding container c009b07d294fbd57de73d2507bbd5376490d331b320f5835a210585eb0b96797: Status 404 returned error can't find the container with id c009b07d294fbd57de73d2507bbd5376490d331b320f5835a210585eb0b96797 Jun 02 01:51:04.687566 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:51:04.687528 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7498df8756-zxx28" event={"ID":"90d23e1c-ca7d-417c-9c8a-136ea2961e30","Type":"ContainerStarted","Data":"c009b07d294fbd57de73d2507bbd5376490d331b320f5835a210585eb0b96797"} Jun 02 01:51:04.688482 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:51:04.688457 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-f99f4b5cd-2pc9n" event={"ID":"ba75ed3b-9f9f-4e81-9665-5310692a5e81","Type":"ContainerStarted","Data":"fe2bdb9eba841b381992a8989ff6d1ebb6eb61b5a710328e908101db7a383039"} Jun 02 01:51:08.703879 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:51:08.703838 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7498df8756-zxx28" event={"ID":"90d23e1c-ca7d-417c-9c8a-136ea2961e30","Type":"ContainerStarted","Data":"8a4cfe2512b6389a8eefded7ef0fb3f1d16525f9b150107bd73136c5b2384bcd"} Jun 02 01:51:08.705222 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:51:08.705196 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-f99f4b5cd-2pc9n" event={"ID":"ba75ed3b-9f9f-4e81-9665-5310692a5e81","Type":"ContainerStarted","Data":"60a53ae6cfab8b3efaae54e95d081eefc62200c783c0626562254a8c75e17d09"} Jun 02 01:51:08.719762 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:51:08.719716 2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/authorino-7498df8756-zxx28" podStartSLOduration=2.27248254 podStartE2EDuration="5.719703414s" podCreationTimestamp="2026-06-02 01:51:03 +0000 UTC" firstStartedPulling="2026-06-02 01:51:04.206318491 +0000 UTC m=+581.852125111" lastFinishedPulling="2026-06-02 01:51:07.653539352 +0000 UTC m=+585.299345985" observedRunningTime="2026-06-02 01:51:08.718031345 +0000 UTC m=+586.363837988" watchObservedRunningTime="2026-06-02 01:51:08.719703414 +0000 UTC m=+586.365510055" Jun 02 01:51:08.734174 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:51:08.734109 2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/authorino-f99f4b5cd-2pc9n" podStartSLOduration=2.128018107 podStartE2EDuration="5.734094657s" podCreationTimestamp="2026-06-02 01:51:03 +0000 UTC" firstStartedPulling="2026-06-02 01:51:04.058402667 +0000 UTC m=+581.704209288" lastFinishedPulling="2026-06-02 01:51:07.664479216 +0000 UTC m=+585.310285838" observedRunningTime="2026-06-02 01:51:08.732455592 +0000 UTC m=+586.378262237" watchObservedRunningTime="2026-06-02 01:51:08.734094657 +0000 UTC m=+586.379901298" Jun 02 01:51:08.765372 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:51:08.765337 2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-f99f4b5cd-2pc9n"] Jun 02 01:51:10.711957 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:51:10.711913 2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="kuadrant-system/authorino-f99f4b5cd-2pc9n" podUID="ba75ed3b-9f9f-4e81-9665-5310692a5e81" containerName="authorino" containerID="cri-o://60a53ae6cfab8b3efaae54e95d081eefc62200c783c0626562254a8c75e17d09" gracePeriod=30 Jun 02 01:51:11.529825 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:51:11.529796 2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f99f4b5cd-2pc9n" Jun 02 01:51:11.680618 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:51:11.680529 2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p9qpc\" (UniqueName: \"kubernetes.io/projected/ba75ed3b-9f9f-4e81-9665-5310692a5e81-kube-api-access-p9qpc\") pod \"ba75ed3b-9f9f-4e81-9665-5310692a5e81\" (UID: \"ba75ed3b-9f9f-4e81-9665-5310692a5e81\") " Jun 02 01:51:11.682721 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:51:11.682688 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba75ed3b-9f9f-4e81-9665-5310692a5e81-kube-api-access-p9qpc" (OuterVolumeSpecName: "kube-api-access-p9qpc") pod "ba75ed3b-9f9f-4e81-9665-5310692a5e81" (UID: "ba75ed3b-9f9f-4e81-9665-5310692a5e81"). InnerVolumeSpecName "kube-api-access-p9qpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 02 01:51:11.716044 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:51:11.716015 2571 generic.go:358] "Generic (PLEG): container finished" podID="ba75ed3b-9f9f-4e81-9665-5310692a5e81" containerID="60a53ae6cfab8b3efaae54e95d081eefc62200c783c0626562254a8c75e17d09" exitCode=0 Jun 02 01:51:11.716450 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:51:11.716072 2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f99f4b5cd-2pc9n" Jun 02 01:51:11.716450 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:51:11.716101 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-f99f4b5cd-2pc9n" event={"ID":"ba75ed3b-9f9f-4e81-9665-5310692a5e81","Type":"ContainerDied","Data":"60a53ae6cfab8b3efaae54e95d081eefc62200c783c0626562254a8c75e17d09"} Jun 02 01:51:11.716450 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:51:11.716160 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-f99f4b5cd-2pc9n" event={"ID":"ba75ed3b-9f9f-4e81-9665-5310692a5e81","Type":"ContainerDied","Data":"fe2bdb9eba841b381992a8989ff6d1ebb6eb61b5a710328e908101db7a383039"} Jun 02 01:51:11.716450 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:51:11.716179 2571 scope.go:117] "RemoveContainer" containerID="60a53ae6cfab8b3efaae54e95d081eefc62200c783c0626562254a8c75e17d09" Jun 02 01:51:11.724227 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:51:11.724207 2571 scope.go:117] "RemoveContainer" containerID="60a53ae6cfab8b3efaae54e95d081eefc62200c783c0626562254a8c75e17d09" Jun 02 01:51:11.724478 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:51:11.724460 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60a53ae6cfab8b3efaae54e95d081eefc62200c783c0626562254a8c75e17d09\": container with ID starting with 60a53ae6cfab8b3efaae54e95d081eefc62200c783c0626562254a8c75e17d09 not found: ID does not exist" containerID="60a53ae6cfab8b3efaae54e95d081eefc62200c783c0626562254a8c75e17d09" Jun 02 01:51:11.724524 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:51:11.724487 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60a53ae6cfab8b3efaae54e95d081eefc62200c783c0626562254a8c75e17d09"} err="failed to get container status \"60a53ae6cfab8b3efaae54e95d081eefc62200c783c0626562254a8c75e17d09\": rpc error: code = NotFound desc = could not find container \"60a53ae6cfab8b3efaae54e95d081eefc62200c783c0626562254a8c75e17d09\": container with ID starting with 60a53ae6cfab8b3efaae54e95d081eefc62200c783c0626562254a8c75e17d09 not found: ID does not exist" Jun 02 01:51:11.737218 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:51:11.737194 2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-f99f4b5cd-2pc9n"] Jun 02 01:51:11.741000 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:51:11.740979 2571 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-f99f4b5cd-2pc9n"] Jun 02 01:51:11.782116 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:51:11.782091 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9qpc\" (UniqueName: \"kubernetes.io/projected/ba75ed3b-9f9f-4e81-9665-5310692a5e81-kube-api-access-p9qpc\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\"" Jun 02 01:51:12.927838 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:51:12.927802 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba75ed3b-9f9f-4e81-9665-5310692a5e81" path="/var/lib/kubelet/pods/ba75ed3b-9f9f-4e81-9665-5310692a5e81/volumes" Jun 02 01:51:52.978868 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:51:52.978836 2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["keycloak-system/maas-keycloak-0"] Jun 02 01:51:52.982246 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:51:52.979173 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba75ed3b-9f9f-4e81-9665-5310692a5e81" containerName="authorino" Jun 02 01:51:52.982246 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:51:52.979187 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba75ed3b-9f9f-4e81-9665-5310692a5e81" containerName="authorino" Jun 02 01:51:52.982246 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:51:52.979257 2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="ba75ed3b-9f9f-4e81-9665-5310692a5e81" containerName="authorino" Jun 02 01:51:52.982246 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:51:52.982215 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="keycloak-system/maas-keycloak-0" Jun 02 01:51:52.984776 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:51:52.984750 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"keycloak-system\"/\"maas-keycloak-initial-admin\"" Jun 02 01:51:52.984882 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:51:52.984790 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"keycloak-system\"/\"openshift-service-ca.crt\"" Jun 02 01:51:52.984882 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:51:52.984790 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"keycloak-system\"/\"kube-root-ca.crt\"" Jun 02 01:51:52.985839 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:51:52.985825 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"keycloak-system\"/\"default-dockercfg-8d6dj\"" Jun 02 01:51:52.989967 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:51:52.989945 2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["keycloak-system/maas-keycloak-0"] Jun 02 01:51:53.026555 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:51:53.026526 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmz7h\" (UniqueName: \"kubernetes.io/projected/b84a29f8-554d-4eb2-ab8e-0608a11f3ad5-kube-api-access-xmz7h\") pod \"maas-keycloak-0\" (UID: \"b84a29f8-554d-4eb2-ab8e-0608a11f3ad5\") " pod="keycloak-system/maas-keycloak-0" Jun 02 01:51:53.127481 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:51:53.127447 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xmz7h\" (UniqueName: \"kubernetes.io/projected/b84a29f8-554d-4eb2-ab8e-0608a11f3ad5-kube-api-access-xmz7h\") pod \"maas-keycloak-0\" (UID: \"b84a29f8-554d-4eb2-ab8e-0608a11f3ad5\") " pod="keycloak-system/maas-keycloak-0" Jun 02 01:51:53.136686 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:51:53.136655 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmz7h\" (UniqueName: \"kubernetes.io/projected/b84a29f8-554d-4eb2-ab8e-0608a11f3ad5-kube-api-access-xmz7h\") pod \"maas-keycloak-0\" (UID: \"b84a29f8-554d-4eb2-ab8e-0608a11f3ad5\") " pod="keycloak-system/maas-keycloak-0" Jun 02 01:51:53.292736 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:51:53.292653 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="keycloak-system/maas-keycloak-0" Jun 02 01:51:53.421432 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:51:53.421408 2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["keycloak-system/maas-keycloak-0"] Jun 02 01:51:53.423716 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:51:53.423691 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb84a29f8_554d_4eb2_ab8e_0608a11f3ad5.slice/crio-df6a838dadf2c0bdf79dd666150e240e1f9c29fe2bd59d58b73149383b7e116b WatchSource:0}: Error finding container df6a838dadf2c0bdf79dd666150e240e1f9c29fe2bd59d58b73149383b7e116b: Status 404 returned error can't find the container with id df6a838dadf2c0bdf79dd666150e240e1f9c29fe2bd59d58b73149383b7e116b Jun 02 01:51:53.425346 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:51:53.425329 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jun 02 01:51:53.846618 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:51:53.846583 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="keycloak-system/maas-keycloak-0" event={"ID":"b84a29f8-554d-4eb2-ab8e-0608a11f3ad5","Type":"ContainerStarted","Data":"df6a838dadf2c0bdf79dd666150e240e1f9c29fe2bd59d58b73149383b7e116b"} Jun 02 01:51:58.865857 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:51:58.865812 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="keycloak-system/maas-keycloak-0" event={"ID":"b84a29f8-554d-4eb2-ab8e-0608a11f3ad5","Type":"ContainerStarted","Data":"7014313f6ac5b5fb1596709e0809ea14ea2200d02818156f0eee7864574f1040"} Jun 02 01:51:58.886080 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:51:58.886018 2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="keycloak-system/maas-keycloak-0" podStartSLOduration=2.090602321 podStartE2EDuration="6.886000761s" podCreationTimestamp="2026-06-02 01:51:52 +0000 UTC" firstStartedPulling="2026-06-02 01:51:53.425446482 +0000 UTC m=+631.071253102" lastFinishedPulling="2026-06-02 01:51:58.220844919 +0000 UTC m=+635.866651542" observedRunningTime="2026-06-02 01:51:58.88303871 +0000 UTC m=+636.528845353" watchObservedRunningTime="2026-06-02 01:51:58.886000761 +0000 UTC m=+636.531807405" Jun 02 01:51:59.293456 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:51:59.293399 2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="keycloak-system/maas-keycloak-0" Jun 02 01:51:59.295052 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:51:59.295013 2571 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="b84a29f8-554d-4eb2-ab8e-0608a11f3ad5" containerName="keycloak" probeResult="failure" output="Get \"http://10.134.0.24:9000/health/started\": dial tcp 10.134.0.24:9000: connect: connection refused" Jun 02 01:52:00.293589 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:00.293531 2571 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="b84a29f8-554d-4eb2-ab8e-0608a11f3ad5" containerName="keycloak" probeResult="failure" output="Get \"http://10.134.0.24:9000/health/started\": dial tcp 10.134.0.24:9000: connect: connection refused" Jun 02 01:52:01.294114 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:01.294057 2571 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="b84a29f8-554d-4eb2-ab8e-0608a11f3ad5" containerName="keycloak" probeResult="failure" output="Get \"http://10.134.0.24:9000/health/started\": dial tcp 10.134.0.24:9000: connect: connection refused" Jun 02 01:52:02.293263 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:02.293195 2571 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="b84a29f8-554d-4eb2-ab8e-0608a11f3ad5" containerName="keycloak" probeResult="failure" output="Get \"http://10.134.0.24:9000/health/started\": dial tcp 10.134.0.24:9000: connect: connection refused" Jun 02 01:52:03.293015 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:03.292978 2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="keycloak-system/maas-keycloak-0" Jun 02 01:52:03.293648 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:03.293235 2571 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="b84a29f8-554d-4eb2-ab8e-0608a11f3ad5" containerName="keycloak" probeResult="failure" output="Get \"http://10.134.0.24:9000/health/started\": dial tcp 10.134.0.24:9000: connect: connection refused" Jun 02 01:52:04.293839 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:04.293792 2571 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="b84a29f8-554d-4eb2-ab8e-0608a11f3ad5" containerName="keycloak" probeResult="failure" output="Get \"http://10.134.0.24:9000/health/started\": dial tcp 10.134.0.24:9000: connect: connection refused" Jun 02 01:52:05.294006 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:05.293957 2571 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="b84a29f8-554d-4eb2-ab8e-0608a11f3ad5" containerName="keycloak" probeResult="failure" output="Get \"http://10.134.0.24:9000/health/started\": dial tcp 10.134.0.24:9000: connect: connection refused" Jun 02 01:52:06.293524 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:06.293476 2571 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="b84a29f8-554d-4eb2-ab8e-0608a11f3ad5" containerName="keycloak" probeResult="failure" output="Get \"http://10.134.0.24:9000/health/started\": dial tcp 10.134.0.24:9000: connect: connection refused" Jun 02 01:52:07.293757 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:07.293711 2571 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="b84a29f8-554d-4eb2-ab8e-0608a11f3ad5" containerName="keycloak" probeResult="failure" output="Get \"http://10.134.0.24:9000/health/started\": dial tcp 10.134.0.24:9000: connect: connection refused" Jun 02 01:52:08.293879 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:08.293824 2571 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="b84a29f8-554d-4eb2-ab8e-0608a11f3ad5" containerName="keycloak" probeResult="failure" output="Get \"http://10.134.0.24:9000/health/started\": dial tcp 10.134.0.24:9000: connect: connection refused" Jun 02 01:52:09.294276 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:09.294216 2571 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="b84a29f8-554d-4eb2-ab8e-0608a11f3ad5" containerName="keycloak" probeResult="failure" output="Get \"http://10.134.0.24:9000/health/started\": dial tcp 10.134.0.24:9000: connect: connection refused" Jun 02 01:52:10.293926 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:10.293873 2571 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="b84a29f8-554d-4eb2-ab8e-0608a11f3ad5" containerName="keycloak" probeResult="failure" output="Get \"http://10.134.0.24:9000/health/started\": dial tcp 10.134.0.24:9000: connect: connection refused" Jun 02 01:52:11.293227 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:11.293183 2571 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="b84a29f8-554d-4eb2-ab8e-0608a11f3ad5" containerName="keycloak" probeResult="failure" output="Get \"http://10.134.0.24:9000/health/started\": dial tcp 10.134.0.24:9000: connect: connection refused" Jun 02 01:52:12.414187 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:12.414126 2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="keycloak-system/maas-keycloak-0" Jun 02 01:52:12.432834 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:12.432776 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="keycloak-system/maas-keycloak-0" podUID="b84a29f8-554d-4eb2-ab8e-0608a11f3ad5" containerName="keycloak" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jun 02 01:52:22.420983 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:22.420948 2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="keycloak-system/maas-keycloak-0" Jun 02 01:52:23.277646 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:23.277613 2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-8b475cf9f-sx9sx"] Jun 02 01:52:23.285523 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:23.285496 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8b475cf9f-sx9sx" Jun 02 01:52:23.285997 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:23.285975 2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8b475cf9f-sx9sx"] Jun 02 01:52:23.325700 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:23.325670 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxq54\" (UniqueName: \"kubernetes.io/projected/1a042064-cd2a-4eb4-85fe-8e2bac8329a8-kube-api-access-gxq54\") pod \"authorino-8b475cf9f-sx9sx\" (UID: \"1a042064-cd2a-4eb4-85fe-8e2bac8329a8\") " pod="kuadrant-system/authorino-8b475cf9f-sx9sx" Jun 02 01:52:23.427054 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:23.427024 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gxq54\" (UniqueName: \"kubernetes.io/projected/1a042064-cd2a-4eb4-85fe-8e2bac8329a8-kube-api-access-gxq54\") pod \"authorino-8b475cf9f-sx9sx\" (UID: \"1a042064-cd2a-4eb4-85fe-8e2bac8329a8\") " pod="kuadrant-system/authorino-8b475cf9f-sx9sx" Jun 02 01:52:23.435829 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:23.435800 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxq54\" (UniqueName: \"kubernetes.io/projected/1a042064-cd2a-4eb4-85fe-8e2bac8329a8-kube-api-access-gxq54\") pod \"authorino-8b475cf9f-sx9sx\" (UID: \"1a042064-cd2a-4eb4-85fe-8e2bac8329a8\") " pod="kuadrant-system/authorino-8b475cf9f-sx9sx" Jun 02 01:52:23.459011 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:23.458986 2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8b475cf9f-sx9sx"] Jun 02 01:52:23.459234 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:23.459222 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8b475cf9f-sx9sx" Jun 02 01:52:23.488824 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:23.488797 2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56fdd757f5-p556q"] Jun 02 01:52:23.491450 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:23.491429 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56fdd757f5-p556q" Jun 02 01:52:23.494100 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:23.494074 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jun 02 01:52:23.499065 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:23.499042 2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56fdd757f5-p556q"] Jun 02 01:52:23.517644 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:23.517620 2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56fdd757f5-p556q"] Jun 02 01:52:23.517890 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:52:23.517871 2571 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-hw5rp tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-56fdd757f5-p556q" podUID="2cd72acc-ecf1-4675-a2c4-eda8bd9a8b91" Jun 02 01:52:23.528359 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:23.528300 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2cd72acc-ecf1-4675-a2c4-eda8bd9a8b91-tls-cert\") pod \"authorino-56fdd757f5-p556q\" (UID: \"2cd72acc-ecf1-4675-a2c4-eda8bd9a8b91\") " pod="kuadrant-system/authorino-56fdd757f5-p556q" Jun 02 01:52:23.528359 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:23.528351 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw5rp\" (UniqueName: \"kubernetes.io/projected/2cd72acc-ecf1-4675-a2c4-eda8bd9a8b91-kube-api-access-hw5rp\") pod \"authorino-56fdd757f5-p556q\" (UID: \"2cd72acc-ecf1-4675-a2c4-eda8bd9a8b91\") " pod="kuadrant-system/authorino-56fdd757f5-p556q" Jun 02 01:52:23.543059 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:23.543034 2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-586bfc7ddf-ccmzj"] Jun 02 01:52:23.548627 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:23.548605 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-586bfc7ddf-ccmzj" Jun 02 01:52:23.555388 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:23.555364 2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-586bfc7ddf-ccmzj"] Jun 02 01:52:23.589610 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:23.589586 2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8b475cf9f-sx9sx"] Jun 02 01:52:23.592033 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:52:23.592004 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1a042064_cd2a_4eb4_85fe_8e2bac8329a8.slice/crio-01d145e343e6bcca6df2d2f32ffe3be4fb1cf4e7a90538c26652870ca9224194 WatchSource:0}: Error finding container 01d145e343e6bcca6df2d2f32ffe3be4fb1cf4e7a90538c26652870ca9224194: Status 404 returned error can't find the container with id 01d145e343e6bcca6df2d2f32ffe3be4fb1cf4e7a90538c26652870ca9224194 Jun 02 01:52:23.629438 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:23.629412 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6fj8\" (UniqueName: \"kubernetes.io/projected/9962c95a-a54c-4096-9d7a-85a9e6b777e7-kube-api-access-v6fj8\") pod \"authorino-586bfc7ddf-ccmzj\" (UID: \"9962c95a-a54c-4096-9d7a-85a9e6b777e7\") " pod="kuadrant-system/authorino-586bfc7ddf-ccmzj" Jun 02 01:52:23.629559 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:23.629468 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2cd72acc-ecf1-4675-a2c4-eda8bd9a8b91-tls-cert\") pod \"authorino-56fdd757f5-p556q\" (UID: \"2cd72acc-ecf1-4675-a2c4-eda8bd9a8b91\") " pod="kuadrant-system/authorino-56fdd757f5-p556q" Jun 02 01:52:23.629559 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:23.629534 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hw5rp\" (UniqueName: \"kubernetes.io/projected/2cd72acc-ecf1-4675-a2c4-eda8bd9a8b91-kube-api-access-hw5rp\") pod \"authorino-56fdd757f5-p556q\" (UID: \"2cd72acc-ecf1-4675-a2c4-eda8bd9a8b91\") " pod="kuadrant-system/authorino-56fdd757f5-p556q" Jun 02 01:52:23.629676 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:23.629571 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9962c95a-a54c-4096-9d7a-85a9e6b777e7-tls-cert\") pod \"authorino-586bfc7ddf-ccmzj\" (UID: \"9962c95a-a54c-4096-9d7a-85a9e6b777e7\") " pod="kuadrant-system/authorino-586bfc7ddf-ccmzj" Jun 02 01:52:23.631787 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:23.631766 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2cd72acc-ecf1-4675-a2c4-eda8bd9a8b91-tls-cert\") pod \"authorino-56fdd757f5-p556q\" (UID: \"2cd72acc-ecf1-4675-a2c4-eda8bd9a8b91\") " pod="kuadrant-system/authorino-56fdd757f5-p556q" Jun 02 01:52:23.637619 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:23.637592 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw5rp\" (UniqueName: \"kubernetes.io/projected/2cd72acc-ecf1-4675-a2c4-eda8bd9a8b91-kube-api-access-hw5rp\") pod \"authorino-56fdd757f5-p556q\" (UID: \"2cd72acc-ecf1-4675-a2c4-eda8bd9a8b91\") " pod="kuadrant-system/authorino-56fdd757f5-p556q" Jun 02 01:52:23.730805 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:23.730777 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-v6fj8\" (UniqueName: \"kubernetes.io/projected/9962c95a-a54c-4096-9d7a-85a9e6b777e7-kube-api-access-v6fj8\") pod \"authorino-586bfc7ddf-ccmzj\" (UID: \"9962c95a-a54c-4096-9d7a-85a9e6b777e7\") " pod="kuadrant-system/authorino-586bfc7ddf-ccmzj" Jun 02 01:52:23.730922 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:23.730862 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9962c95a-a54c-4096-9d7a-85a9e6b777e7-tls-cert\") pod \"authorino-586bfc7ddf-ccmzj\" (UID: \"9962c95a-a54c-4096-9d7a-85a9e6b777e7\") " pod="kuadrant-system/authorino-586bfc7ddf-ccmzj" Jun 02 01:52:23.733098 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:23.733076 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9962c95a-a54c-4096-9d7a-85a9e6b777e7-tls-cert\") pod \"authorino-586bfc7ddf-ccmzj\" (UID: \"9962c95a-a54c-4096-9d7a-85a9e6b777e7\") " pod="kuadrant-system/authorino-586bfc7ddf-ccmzj" Jun 02 01:52:23.737676 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:23.737652 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6fj8\" (UniqueName: \"kubernetes.io/projected/9962c95a-a54c-4096-9d7a-85a9e6b777e7-kube-api-access-v6fj8\") pod \"authorino-586bfc7ddf-ccmzj\" (UID: \"9962c95a-a54c-4096-9d7a-85a9e6b777e7\") " pod="kuadrant-system/authorino-586bfc7ddf-ccmzj" Jun 02 01:52:23.860155 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:23.860068 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-586bfc7ddf-ccmzj" Jun 02 01:52:23.961012 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:23.960984 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8b475cf9f-sx9sx" event={"ID":"1a042064-cd2a-4eb4-85fe-8e2bac8329a8","Type":"ContainerStarted","Data":"01d145e343e6bcca6df2d2f32ffe3be4fb1cf4e7a90538c26652870ca9224194"} Jun 02 01:52:23.961146 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:23.961019 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56fdd757f5-p556q" Jun 02 01:52:23.972759 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:23.972740 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56fdd757f5-p556q" Jun 02 01:52:23.981703 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:23.981681 2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-586bfc7ddf-ccmzj"] Jun 02 01:52:23.983871 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:52:23.983850 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9962c95a_a54c_4096_9d7a_85a9e6b777e7.slice/crio-83a521d44c73d4e7eb9ba985e5b702d8e5c15981d3048e0d238c1fad76f8b6f9 WatchSource:0}: Error finding container 83a521d44c73d4e7eb9ba985e5b702d8e5c15981d3048e0d238c1fad76f8b6f9: Status 404 returned error can't find the container with id 83a521d44c73d4e7eb9ba985e5b702d8e5c15981d3048e0d238c1fad76f8b6f9 Jun 02 01:52:24.033911 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:24.033876 2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hw5rp\" (UniqueName: \"kubernetes.io/projected/2cd72acc-ecf1-4675-a2c4-eda8bd9a8b91-kube-api-access-hw5rp\") pod \"2cd72acc-ecf1-4675-a2c4-eda8bd9a8b91\" (UID: \"2cd72acc-ecf1-4675-a2c4-eda8bd9a8b91\") " Jun 02 01:52:24.034306 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:24.033950 2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2cd72acc-ecf1-4675-a2c4-eda8bd9a8b91-tls-cert\") pod \"2cd72acc-ecf1-4675-a2c4-eda8bd9a8b91\" (UID: \"2cd72acc-ecf1-4675-a2c4-eda8bd9a8b91\") " Jun 02 01:52:24.036404 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:24.036373 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cd72acc-ecf1-4675-a2c4-eda8bd9a8b91-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "2cd72acc-ecf1-4675-a2c4-eda8bd9a8b91" (UID: "2cd72acc-ecf1-4675-a2c4-eda8bd9a8b91"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 02 01:52:24.036511 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:24.036474 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cd72acc-ecf1-4675-a2c4-eda8bd9a8b91-kube-api-access-hw5rp" (OuterVolumeSpecName: "kube-api-access-hw5rp") pod "2cd72acc-ecf1-4675-a2c4-eda8bd9a8b91" (UID: "2cd72acc-ecf1-4675-a2c4-eda8bd9a8b91"). InnerVolumeSpecName "kube-api-access-hw5rp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 02 01:52:24.135662 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:24.135583 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hw5rp\" (UniqueName: \"kubernetes.io/projected/2cd72acc-ecf1-4675-a2c4-eda8bd9a8b91-kube-api-access-hw5rp\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\"" Jun 02 01:52:24.135662 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:24.135618 2571 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2cd72acc-ecf1-4675-a2c4-eda8bd9a8b91-tls-cert\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\"" Jun 02 01:52:24.965459 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:24.965423 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8b475cf9f-sx9sx" event={"ID":"1a042064-cd2a-4eb4-85fe-8e2bac8329a8","Type":"ContainerStarted","Data":"a74b5697b93ee3d092dcb8f593e280d16e7298a7170ba97b149884895fc54263"} Jun 02 01:52:24.965883 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:24.965503 2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="kuadrant-system/authorino-8b475cf9f-sx9sx" podUID="1a042064-cd2a-4eb4-85fe-8e2bac8329a8" containerName="authorino" containerID="cri-o://a74b5697b93ee3d092dcb8f593e280d16e7298a7170ba97b149884895fc54263" gracePeriod=30 Jun 02 01:52:24.966838 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:24.966806 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-586bfc7ddf-ccmzj" event={"ID":"9962c95a-a54c-4096-9d7a-85a9e6b777e7","Type":"ContainerStarted","Data":"c5ab2a058f7589aeffd134e7f738ed0dcf90719df6a96e8f20075dde80a5e46d"} Jun 02 01:52:24.967005 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:24.966842 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56fdd757f5-p556q" Jun 02 01:52:24.967005 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:24.966845 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-586bfc7ddf-ccmzj" event={"ID":"9962c95a-a54c-4096-9d7a-85a9e6b777e7","Type":"ContainerStarted","Data":"83a521d44c73d4e7eb9ba985e5b702d8e5c15981d3048e0d238c1fad76f8b6f9"} Jun 02 01:52:24.983644 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:24.983603 2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/authorino-8b475cf9f-sx9sx" podStartSLOduration=1.634866831 podStartE2EDuration="1.983591113s" podCreationTimestamp="2026-06-02 01:52:23 +0000 UTC" firstStartedPulling="2026-06-02 01:52:23.593464947 +0000 UTC m=+661.239271570" lastFinishedPulling="2026-06-02 01:52:23.942189221 +0000 UTC m=+661.587995852" observedRunningTime="2026-06-02 01:52:24.982057009 +0000 UTC m=+662.627863652" watchObservedRunningTime="2026-06-02 01:52:24.983591113 +0000 UTC m=+662.629397756" Jun 02 01:52:24.997519 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:24.997474 2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/authorino-586bfc7ddf-ccmzj" podStartSLOduration=1.715125836 podStartE2EDuration="1.99746295s" podCreationTimestamp="2026-06-02 01:52:23 +0000 UTC" firstStartedPulling="2026-06-02 01:52:23.985197057 +0000 UTC m=+661.631003678" lastFinishedPulling="2026-06-02 01:52:24.267534168 +0000 UTC m=+661.913340792" observedRunningTime="2026-06-02 01:52:24.996566594 +0000 UTC m=+662.642373239" watchObservedRunningTime="2026-06-02 01:52:24.99746295 +0000 UTC m=+662.643269592" Jun 02 01:52:25.021614 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:25.021584 2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56fdd757f5-p556q"] Jun 02 01:52:25.026600 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:25.026563 2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7498df8756-zxx28"] Jun 02 01:52:25.026807 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:25.026784 2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="kuadrant-system/authorino-7498df8756-zxx28" podUID="90d23e1c-ca7d-417c-9c8a-136ea2961e30" containerName="authorino" containerID="cri-o://8a4cfe2512b6389a8eefded7ef0fb3f1d16525f9b150107bd73136c5b2384bcd" gracePeriod=30 Jun 02 01:52:25.030028 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:25.030000 2571 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56fdd757f5-p556q"] Jun 02 01:52:25.270389 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:25.270365 2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8b475cf9f-sx9sx" Jun 02 01:52:25.273432 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:25.273411 2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7498df8756-zxx28" Jun 02 01:52:25.344558 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:25.344520 2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gxq54\" (UniqueName: \"kubernetes.io/projected/1a042064-cd2a-4eb4-85fe-8e2bac8329a8-kube-api-access-gxq54\") pod \"1a042064-cd2a-4eb4-85fe-8e2bac8329a8\" (UID: \"1a042064-cd2a-4eb4-85fe-8e2bac8329a8\") " Jun 02 01:52:25.344716 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:25.344616 2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8v7mh\" (UniqueName: \"kubernetes.io/projected/90d23e1c-ca7d-417c-9c8a-136ea2961e30-kube-api-access-8v7mh\") pod \"90d23e1c-ca7d-417c-9c8a-136ea2961e30\" (UID: \"90d23e1c-ca7d-417c-9c8a-136ea2961e30\") " Jun 02 01:52:25.346663 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:25.346631 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a042064-cd2a-4eb4-85fe-8e2bac8329a8-kube-api-access-gxq54" (OuterVolumeSpecName: "kube-api-access-gxq54") pod "1a042064-cd2a-4eb4-85fe-8e2bac8329a8" (UID: "1a042064-cd2a-4eb4-85fe-8e2bac8329a8"). InnerVolumeSpecName "kube-api-access-gxq54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 02 01:52:25.346663 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:25.346653 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90d23e1c-ca7d-417c-9c8a-136ea2961e30-kube-api-access-8v7mh" (OuterVolumeSpecName: "kube-api-access-8v7mh") pod "90d23e1c-ca7d-417c-9c8a-136ea2961e30" (UID: "90d23e1c-ca7d-417c-9c8a-136ea2961e30"). InnerVolumeSpecName "kube-api-access-8v7mh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 02 01:52:25.445994 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:25.445970 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8v7mh\" (UniqueName: \"kubernetes.io/projected/90d23e1c-ca7d-417c-9c8a-136ea2961e30-kube-api-access-8v7mh\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\"" Jun 02 01:52:25.445994 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:25.445993 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxq54\" (UniqueName: \"kubernetes.io/projected/1a042064-cd2a-4eb4-85fe-8e2bac8329a8-kube-api-access-gxq54\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\"" Jun 02 01:52:25.970622 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:25.970583 2571 generic.go:358] "Generic (PLEG): container finished" podID="90d23e1c-ca7d-417c-9c8a-136ea2961e30" containerID="8a4cfe2512b6389a8eefded7ef0fb3f1d16525f9b150107bd73136c5b2384bcd" exitCode=0 Jun 02 01:52:25.971073 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:25.970631 2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7498df8756-zxx28" Jun 02 01:52:25.971073 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:25.970649 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7498df8756-zxx28" event={"ID":"90d23e1c-ca7d-417c-9c8a-136ea2961e30","Type":"ContainerDied","Data":"8a4cfe2512b6389a8eefded7ef0fb3f1d16525f9b150107bd73136c5b2384bcd"} Jun 02 01:52:25.971073 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:25.970679 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7498df8756-zxx28" event={"ID":"90d23e1c-ca7d-417c-9c8a-136ea2961e30","Type":"ContainerDied","Data":"c009b07d294fbd57de73d2507bbd5376490d331b320f5835a210585eb0b96797"} Jun 02 01:52:25.971073 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:25.970699 2571 scope.go:117] "RemoveContainer" containerID="8a4cfe2512b6389a8eefded7ef0fb3f1d16525f9b150107bd73136c5b2384bcd" Jun 02 01:52:25.971835 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:25.971799 2571 generic.go:358] "Generic (PLEG): container finished" podID="1a042064-cd2a-4eb4-85fe-8e2bac8329a8" containerID="a74b5697b93ee3d092dcb8f593e280d16e7298a7170ba97b149884895fc54263" exitCode=0 Jun 02 01:52:25.971885 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:25.971848 2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8b475cf9f-sx9sx" Jun 02 01:52:25.971943 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:25.971844 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8b475cf9f-sx9sx" event={"ID":"1a042064-cd2a-4eb4-85fe-8e2bac8329a8","Type":"ContainerDied","Data":"a74b5697b93ee3d092dcb8f593e280d16e7298a7170ba97b149884895fc54263"} Jun 02 01:52:25.971976 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:25.971956 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8b475cf9f-sx9sx" event={"ID":"1a042064-cd2a-4eb4-85fe-8e2bac8329a8","Type":"ContainerDied","Data":"01d145e343e6bcca6df2d2f32ffe3be4fb1cf4e7a90538c26652870ca9224194"} Jun 02 01:52:25.979372 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:25.979356 2571 scope.go:117] "RemoveContainer" containerID="8a4cfe2512b6389a8eefded7ef0fb3f1d16525f9b150107bd73136c5b2384bcd" Jun 02 01:52:25.979640 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:52:25.979620 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a4cfe2512b6389a8eefded7ef0fb3f1d16525f9b150107bd73136c5b2384bcd\": container with ID starting with 8a4cfe2512b6389a8eefded7ef0fb3f1d16525f9b150107bd73136c5b2384bcd not found: ID does not exist" containerID="8a4cfe2512b6389a8eefded7ef0fb3f1d16525f9b150107bd73136c5b2384bcd" Jun 02 01:52:25.979713 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:25.979651 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a4cfe2512b6389a8eefded7ef0fb3f1d16525f9b150107bd73136c5b2384bcd"} err="failed to get container status \"8a4cfe2512b6389a8eefded7ef0fb3f1d16525f9b150107bd73136c5b2384bcd\": rpc error: code = NotFound desc = could not find container \"8a4cfe2512b6389a8eefded7ef0fb3f1d16525f9b150107bd73136c5b2384bcd\": container with ID starting with 8a4cfe2512b6389a8eefded7ef0fb3f1d16525f9b150107bd73136c5b2384bcd not found: ID does not exist" Jun 02 01:52:25.979713 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:25.979676 2571 scope.go:117] "RemoveContainer" containerID="a74b5697b93ee3d092dcb8f593e280d16e7298a7170ba97b149884895fc54263" Jun 02 01:52:25.987612 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:25.987598 2571 scope.go:117] "RemoveContainer" containerID="a74b5697b93ee3d092dcb8f593e280d16e7298a7170ba97b149884895fc54263" Jun 02 01:52:25.987843 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:52:25.987821 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a74b5697b93ee3d092dcb8f593e280d16e7298a7170ba97b149884895fc54263\": container with ID starting with a74b5697b93ee3d092dcb8f593e280d16e7298a7170ba97b149884895fc54263 not found: ID does not exist" containerID="a74b5697b93ee3d092dcb8f593e280d16e7298a7170ba97b149884895fc54263" Jun 02 01:52:25.987922 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:25.987847 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a74b5697b93ee3d092dcb8f593e280d16e7298a7170ba97b149884895fc54263"} err="failed to get container status \"a74b5697b93ee3d092dcb8f593e280d16e7298a7170ba97b149884895fc54263\": rpc error: code = NotFound desc = could not find container \"a74b5697b93ee3d092dcb8f593e280d16e7298a7170ba97b149884895fc54263\": container with ID starting with a74b5697b93ee3d092dcb8f593e280d16e7298a7170ba97b149884895fc54263 not found: ID does not exist" Jun 02 01:52:25.993969 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:25.993947 2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7498df8756-zxx28"] Jun 02 01:52:25.999325 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:25.999303 2571 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7498df8756-zxx28"] Jun 02 01:52:26.009819 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:26.009797 2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8b475cf9f-sx9sx"] Jun 02 01:52:26.013018 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:26.012997 2571 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-8b475cf9f-sx9sx"] Jun 02 01:52:26.928074 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:26.928038 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a042064-cd2a-4eb4-85fe-8e2bac8329a8" path="/var/lib/kubelet/pods/1a042064-cd2a-4eb4-85fe-8e2bac8329a8/volumes" Jun 02 01:52:26.928429 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:26.928415 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2cd72acc-ecf1-4675-a2c4-eda8bd9a8b91" path="/var/lib/kubelet/pods/2cd72acc-ecf1-4675-a2c4-eda8bd9a8b91/volumes" Jun 02 01:52:26.928630 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:26.928619 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90d23e1c-ca7d-417c-9c8a-136ea2961e30" path="/var/lib/kubelet/pods/90d23e1c-ca7d-417c-9c8a-136ea2961e30/volumes" Jun 02 01:52:28.106247 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:28.106212 2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-694d9bcf7c-bnjzn"] Jun 02 01:52:28.106608 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:28.106547 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90d23e1c-ca7d-417c-9c8a-136ea2961e30" containerName="authorino" Jun 02 01:52:28.106608 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:28.106559 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="90d23e1c-ca7d-417c-9c8a-136ea2961e30" containerName="authorino" Jun 02 01:52:28.106608 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:28.106572 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a042064-cd2a-4eb4-85fe-8e2bac8329a8" containerName="authorino" Jun 02 01:52:28.106608 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:28.106578 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a042064-cd2a-4eb4-85fe-8e2bac8329a8" containerName="authorino" Jun 02 01:52:28.106765 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:28.106635 2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="90d23e1c-ca7d-417c-9c8a-136ea2961e30" containerName="authorino" Jun 02 01:52:28.106765 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:28.106647 2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="1a042064-cd2a-4eb4-85fe-8e2bac8329a8" containerName="authorino" Jun 02 01:52:28.109349 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:28.109331 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-694d9bcf7c-bnjzn" Jun 02 01:52:28.111919 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:28.111896 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jun 02 01:52:28.112037 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:28.111926 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-9kv9q\"" Jun 02 01:52:28.118563 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:28.118541 2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-694d9bcf7c-bnjzn"] Jun 02 01:52:28.169430 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:28.169393 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtbff\" (UniqueName: \"kubernetes.io/projected/66a12b14-4dd6-4281-a1fb-587484867514-kube-api-access-mtbff\") pod \"maas-controller-694d9bcf7c-bnjzn\" (UID: \"66a12b14-4dd6-4281-a1fb-587484867514\") " pod="opendatahub/maas-controller-694d9bcf7c-bnjzn" Jun 02 01:52:28.270562 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:28.270526 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mtbff\" (UniqueName: \"kubernetes.io/projected/66a12b14-4dd6-4281-a1fb-587484867514-kube-api-access-mtbff\") pod \"maas-controller-694d9bcf7c-bnjzn\" (UID: \"66a12b14-4dd6-4281-a1fb-587484867514\") " pod="opendatahub/maas-controller-694d9bcf7c-bnjzn" Jun 02 01:52:28.292311 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:28.292282 2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7f759c8d56-bdxj7"] Jun 02 01:52:28.294593 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:28.294577 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f759c8d56-bdxj7" Jun 02 01:52:28.300612 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:28.300583 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtbff\" (UniqueName: \"kubernetes.io/projected/66a12b14-4dd6-4281-a1fb-587484867514-kube-api-access-mtbff\") pod \"maas-controller-694d9bcf7c-bnjzn\" (UID: \"66a12b14-4dd6-4281-a1fb-587484867514\") " pod="opendatahub/maas-controller-694d9bcf7c-bnjzn" Jun 02 01:52:28.314555 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:28.314529 2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7f759c8d56-bdxj7"] Jun 02 01:52:28.371810 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:28.371740 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55rpx\" (UniqueName: \"kubernetes.io/projected/30029d1b-9ffd-42d2-963a-a7ce6eee50b0-kube-api-access-55rpx\") pod \"maas-controller-7f759c8d56-bdxj7\" (UID: \"30029d1b-9ffd-42d2-963a-a7ce6eee50b0\") " pod="opendatahub/maas-controller-7f759c8d56-bdxj7" Jun 02 01:52:28.419691 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:28.419663 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-694d9bcf7c-bnjzn" Jun 02 01:52:28.472908 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:28.472875 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-55rpx\" (UniqueName: \"kubernetes.io/projected/30029d1b-9ffd-42d2-963a-a7ce6eee50b0-kube-api-access-55rpx\") pod \"maas-controller-7f759c8d56-bdxj7\" (UID: \"30029d1b-9ffd-42d2-963a-a7ce6eee50b0\") " pod="opendatahub/maas-controller-7f759c8d56-bdxj7" Jun 02 01:52:28.482014 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:28.481984 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55rpx\" (UniqueName: \"kubernetes.io/projected/30029d1b-9ffd-42d2-963a-a7ce6eee50b0-kube-api-access-55rpx\") pod \"maas-controller-7f759c8d56-bdxj7\" (UID: \"30029d1b-9ffd-42d2-963a-a7ce6eee50b0\") " pod="opendatahub/maas-controller-7f759c8d56-bdxj7" Jun 02 01:52:28.546881 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:28.546849 2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-694d9bcf7c-bnjzn"] Jun 02 01:52:28.550624 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:52:28.550599 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66a12b14_4dd6_4281_a1fb_587484867514.slice/crio-3c4efe5769259b7384eb244ecb3e76cf5b0e320830bc2ac41238e38cca3f4413 WatchSource:0}: Error finding container 3c4efe5769259b7384eb244ecb3e76cf5b0e320830bc2ac41238e38cca3f4413: Status 404 returned error can't find the container with id 3c4efe5769259b7384eb244ecb3e76cf5b0e320830bc2ac41238e38cca3f4413 Jun 02 01:52:28.611224 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:28.611184 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f759c8d56-bdxj7" Jun 02 01:52:28.733739 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:28.733717 2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7f759c8d56-bdxj7"] Jun 02 01:52:28.735721 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:52:28.735693 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30029d1b_9ffd_42d2_963a_a7ce6eee50b0.slice/crio-ccf36be4ed15e61713d53bc0f4b062cfefedcc294ad9b9a3004f537efd19f678 WatchSource:0}: Error finding container ccf36be4ed15e61713d53bc0f4b062cfefedcc294ad9b9a3004f537efd19f678: Status 404 returned error can't find the container with id ccf36be4ed15e61713d53bc0f4b062cfefedcc294ad9b9a3004f537efd19f678 Jun 02 01:52:28.984541 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:28.984456 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-694d9bcf7c-bnjzn" event={"ID":"66a12b14-4dd6-4281-a1fb-587484867514","Type":"ContainerStarted","Data":"3c4efe5769259b7384eb244ecb3e76cf5b0e320830bc2ac41238e38cca3f4413"} Jun 02 01:52:28.985335 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:28.985309 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f759c8d56-bdxj7" event={"ID":"30029d1b-9ffd-42d2-963a-a7ce6eee50b0","Type":"ContainerStarted","Data":"ccf36be4ed15e61713d53bc0f4b062cfefedcc294ad9b9a3004f537efd19f678"} Jun 02 01:52:31.997913 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:31.997868 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f759c8d56-bdxj7" event={"ID":"30029d1b-9ffd-42d2-963a-a7ce6eee50b0","Type":"ContainerStarted","Data":"5cea098ea67c3d972dfa2da03650b7fb54d1c2e6164e946d694360058b87a234"} Jun 02 01:52:31.998392 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:31.997941 2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7f759c8d56-bdxj7" Jun 02 01:52:31.999302 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:31.999276 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-694d9bcf7c-bnjzn" event={"ID":"66a12b14-4dd6-4281-a1fb-587484867514","Type":"ContainerStarted","Data":"61ca7cff395230ec376650426e67760fef4a5abc2bf323557d24846a41ae6e26"} Jun 02 01:52:31.999442 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:31.999386 2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-694d9bcf7c-bnjzn" Jun 02 01:52:32.012701 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:32.012620 2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="opendatahub/maas-controller-7f759c8d56-bdxj7" podStartSLOduration=1.4960149139999999 podStartE2EDuration="4.012604191s" podCreationTimestamp="2026-06-02 01:52:28 +0000 UTC" firstStartedPulling="2026-06-02 01:52:28.736970163 +0000 UTC m=+666.382776786" lastFinishedPulling="2026-06-02 01:52:31.253559422 +0000 UTC m=+668.899366063" observedRunningTime="2026-06-02 01:52:32.012184957 +0000 UTC m=+669.657991600" watchObservedRunningTime="2026-06-02 01:52:32.012604191 +0000 UTC m=+669.658410834" Jun 02 01:52:32.026872 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:32.026822 2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="opendatahub/maas-controller-694d9bcf7c-bnjzn" podStartSLOduration=1.33556317 podStartE2EDuration="4.026807456s" podCreationTimestamp="2026-06-02 01:52:28 +0000 UTC" firstStartedPulling="2026-06-02 01:52:28.552329767 +0000 UTC m=+666.198136388" lastFinishedPulling="2026-06-02 01:52:31.243574047 +0000 UTC m=+668.889380674" observedRunningTime="2026-06-02 01:52:32.025703993 +0000 UTC m=+669.671510634" watchObservedRunningTime="2026-06-02 01:52:32.026807456 +0000 UTC m=+669.672614097" Jun 02 01:52:43.007903 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:43.007868 2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-694d9bcf7c-bnjzn" Jun 02 01:52:43.008306 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:43.007994 2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7f759c8d56-bdxj7" Jun 02 01:52:43.066298 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:43.066259 2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-694d9bcf7c-bnjzn"] Jun 02 01:52:43.066533 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:43.066491 2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="opendatahub/maas-controller-694d9bcf7c-bnjzn" podUID="66a12b14-4dd6-4281-a1fb-587484867514" containerName="manager" containerID="cri-o://61ca7cff395230ec376650426e67760fef4a5abc2bf323557d24846a41ae6e26" gracePeriod=30 Jun 02 01:52:43.310403 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:43.310378 2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-694d9bcf7c-bnjzn" Jun 02 01:52:43.409922 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:43.409883 2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mtbff\" (UniqueName: \"kubernetes.io/projected/66a12b14-4dd6-4281-a1fb-587484867514-kube-api-access-mtbff\") pod \"66a12b14-4dd6-4281-a1fb-587484867514\" (UID: \"66a12b14-4dd6-4281-a1fb-587484867514\") " Jun 02 01:52:43.412117 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:43.412087 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66a12b14-4dd6-4281-a1fb-587484867514-kube-api-access-mtbff" (OuterVolumeSpecName: "kube-api-access-mtbff") pod "66a12b14-4dd6-4281-a1fb-587484867514" (UID: "66a12b14-4dd6-4281-a1fb-587484867514"). InnerVolumeSpecName "kube-api-access-mtbff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 02 01:52:43.511293 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:43.511260 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtbff\" (UniqueName: \"kubernetes.io/projected/66a12b14-4dd6-4281-a1fb-587484867514-kube-api-access-mtbff\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\"" Jun 02 01:52:44.039829 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:44.039793 2571 generic.go:358] "Generic (PLEG): container finished" podID="66a12b14-4dd6-4281-a1fb-587484867514" containerID="61ca7cff395230ec376650426e67760fef4a5abc2bf323557d24846a41ae6e26" exitCode=0 Jun 02 01:52:44.040286 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:44.039860 2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-694d9bcf7c-bnjzn" Jun 02 01:52:44.040286 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:44.039880 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-694d9bcf7c-bnjzn" event={"ID":"66a12b14-4dd6-4281-a1fb-587484867514","Type":"ContainerDied","Data":"61ca7cff395230ec376650426e67760fef4a5abc2bf323557d24846a41ae6e26"} Jun 02 01:52:44.040286 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:44.039925 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-694d9bcf7c-bnjzn" event={"ID":"66a12b14-4dd6-4281-a1fb-587484867514","Type":"ContainerDied","Data":"3c4efe5769259b7384eb244ecb3e76cf5b0e320830bc2ac41238e38cca3f4413"} Jun 02 01:52:44.040286 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:44.039948 2571 scope.go:117] "RemoveContainer" containerID="61ca7cff395230ec376650426e67760fef4a5abc2bf323557d24846a41ae6e26" Jun 02 01:52:44.048111 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:44.048095 2571 scope.go:117] "RemoveContainer" containerID="61ca7cff395230ec376650426e67760fef4a5abc2bf323557d24846a41ae6e26" Jun 02 01:52:44.048375 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:52:44.048355 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61ca7cff395230ec376650426e67760fef4a5abc2bf323557d24846a41ae6e26\": container with ID starting with 61ca7cff395230ec376650426e67760fef4a5abc2bf323557d24846a41ae6e26 not found: ID does not exist" containerID="61ca7cff395230ec376650426e67760fef4a5abc2bf323557d24846a41ae6e26" Jun 02 01:52:44.048427 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:44.048386 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61ca7cff395230ec376650426e67760fef4a5abc2bf323557d24846a41ae6e26"} err="failed to get container status \"61ca7cff395230ec376650426e67760fef4a5abc2bf323557d24846a41ae6e26\": rpc error: code = NotFound desc = could not find container \"61ca7cff395230ec376650426e67760fef4a5abc2bf323557d24846a41ae6e26\": container with ID starting with 61ca7cff395230ec376650426e67760fef4a5abc2bf323557d24846a41ae6e26 not found: ID does not exist" Jun 02 01:52:44.059971 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:44.059949 2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-694d9bcf7c-bnjzn"] Jun 02 01:52:44.064661 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:44.064639 2571 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-694d9bcf7c-bnjzn"] Jun 02 01:52:44.927558 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:44.927526 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66a12b14-4dd6-4281-a1fb-587484867514" path="/var/lib/kubelet/pods/66a12b14-4dd6-4281-a1fb-587484867514/volumes" Jun 02 01:52:59.667657 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:59.667619 2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-6b989f9fdb-nfrzb"] Jun 02 01:52:59.668054 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:59.667961 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66a12b14-4dd6-4281-a1fb-587484867514" containerName="manager" Jun 02 01:52:59.668054 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:59.667974 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="66a12b14-4dd6-4281-a1fb-587484867514" containerName="manager" Jun 02 01:52:59.668054 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:59.668045 2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="66a12b14-4dd6-4281-a1fb-587484867514" containerName="manager" Jun 02 01:52:59.672343 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:59.672324 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-6b989f9fdb-nfrzb" Jun 02 01:52:59.674921 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:59.674884 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-serving-cert\"" Jun 02 01:52:59.675052 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:59.674986 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-85n2b\"" Jun 02 01:52:59.681695 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:59.681667 2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-6b989f9fdb-nfrzb"] Jun 02 01:52:59.744757 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:59.744714 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3948f648-01e8-47e1-a1fc-e531b48f601b-maas-api-tls\") pod \"maas-api-6b989f9fdb-nfrzb\" (UID: \"3948f648-01e8-47e1-a1fc-e531b48f601b\") " pod="opendatahub/maas-api-6b989f9fdb-nfrzb" Jun 02 01:52:59.744757 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:59.744761 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dppfx\" (UniqueName: \"kubernetes.io/projected/3948f648-01e8-47e1-a1fc-e531b48f601b-kube-api-access-dppfx\") pod \"maas-api-6b989f9fdb-nfrzb\" (UID: \"3948f648-01e8-47e1-a1fc-e531b48f601b\") " pod="opendatahub/maas-api-6b989f9fdb-nfrzb" Jun 02 01:52:59.846226 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:59.846187 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3948f648-01e8-47e1-a1fc-e531b48f601b-maas-api-tls\") pod \"maas-api-6b989f9fdb-nfrzb\" (UID: \"3948f648-01e8-47e1-a1fc-e531b48f601b\") " pod="opendatahub/maas-api-6b989f9fdb-nfrzb" Jun 02 01:52:59.846226 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:59.846228 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dppfx\" (UniqueName: \"kubernetes.io/projected/3948f648-01e8-47e1-a1fc-e531b48f601b-kube-api-access-dppfx\") pod \"maas-api-6b989f9fdb-nfrzb\" (UID: \"3948f648-01e8-47e1-a1fc-e531b48f601b\") " pod="opendatahub/maas-api-6b989f9fdb-nfrzb" Jun 02 01:52:59.846497 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:52:59.846351 2571 secret.go:189] Couldn't get secret opendatahub/maas-api-serving-cert: secret "maas-api-serving-cert" not found Jun 02 01:52:59.846497 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:52:59.846426 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3948f648-01e8-47e1-a1fc-e531b48f601b-maas-api-tls podName:3948f648-01e8-47e1-a1fc-e531b48f601b nodeName:}" failed. No retries permitted until 2026-06-02 01:53:00.346405572 +0000 UTC m=+697.992212193 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3948f648-01e8-47e1-a1fc-e531b48f601b-maas-api-tls") pod "maas-api-6b989f9fdb-nfrzb" (UID: "3948f648-01e8-47e1-a1fc-e531b48f601b") : secret "maas-api-serving-cert" not found Jun 02 01:52:59.855502 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:52:59.855470 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dppfx\" (UniqueName: \"kubernetes.io/projected/3948f648-01e8-47e1-a1fc-e531b48f601b-kube-api-access-dppfx\") pod \"maas-api-6b989f9fdb-nfrzb\" (UID: \"3948f648-01e8-47e1-a1fc-e531b48f601b\") " pod="opendatahub/maas-api-6b989f9fdb-nfrzb" Jun 02 01:53:00.350395 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:00.350352 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3948f648-01e8-47e1-a1fc-e531b48f601b-maas-api-tls\") pod \"maas-api-6b989f9fdb-nfrzb\" (UID: \"3948f648-01e8-47e1-a1fc-e531b48f601b\") " pod="opendatahub/maas-api-6b989f9fdb-nfrzb" Jun 02 01:53:00.350563 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:53:00.350531 2571 secret.go:189] Couldn't get secret opendatahub/maas-api-serving-cert: secret "maas-api-serving-cert" not found Jun 02 01:53:00.350626 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:53:00.350614 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3948f648-01e8-47e1-a1fc-e531b48f601b-maas-api-tls podName:3948f648-01e8-47e1-a1fc-e531b48f601b nodeName:}" failed. No retries permitted until 2026-06-02 01:53:01.350593231 +0000 UTC m=+698.996399855 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3948f648-01e8-47e1-a1fc-e531b48f601b-maas-api-tls") pod "maas-api-6b989f9fdb-nfrzb" (UID: "3948f648-01e8-47e1-a1fc-e531b48f601b") : secret "maas-api-serving-cert" not found Jun 02 01:53:01.359158 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:01.359092 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3948f648-01e8-47e1-a1fc-e531b48f601b-maas-api-tls\") pod \"maas-api-6b989f9fdb-nfrzb\" (UID: \"3948f648-01e8-47e1-a1fc-e531b48f601b\") " pod="opendatahub/maas-api-6b989f9fdb-nfrzb" Jun 02 01:53:01.361641 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:01.361618 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3948f648-01e8-47e1-a1fc-e531b48f601b-maas-api-tls\") pod \"maas-api-6b989f9fdb-nfrzb\" (UID: \"3948f648-01e8-47e1-a1fc-e531b48f601b\") " pod="opendatahub/maas-api-6b989f9fdb-nfrzb" Jun 02 01:53:01.483314 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:01.483265 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-6b989f9fdb-nfrzb" Jun 02 01:53:01.621675 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:01.621629 2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-6b989f9fdb-nfrzb"] Jun 02 01:53:01.624057 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:53:01.624027 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3948f648_01e8_47e1_a1fc_e531b48f601b.slice/crio-266f6733c6550378021907c29ff25a6764911c32bed7fe9aef138e9c66a94418 WatchSource:0}: Error finding container 266f6733c6550378021907c29ff25a6764911c32bed7fe9aef138e9c66a94418: Status 404 returned error can't find the container with id 266f6733c6550378021907c29ff25a6764911c32bed7fe9aef138e9c66a94418 Jun 02 01:53:02.097306 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:02.097272 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-6b989f9fdb-nfrzb" event={"ID":"3948f648-01e8-47e1-a1fc-e531b48f601b","Type":"ContainerStarted","Data":"266f6733c6550378021907c29ff25a6764911c32bed7fe9aef138e9c66a94418"} Jun 02 01:53:04.105250 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:04.105167 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-6b989f9fdb-nfrzb" event={"ID":"3948f648-01e8-47e1-a1fc-e531b48f601b","Type":"ContainerStarted","Data":"d93286e85035a553dfbe9449730b508325fd838cfa57eec1c4d54a47589ce472"} Jun 02 01:53:04.105600 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:04.105292 2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-6b989f9fdb-nfrzb" Jun 02 01:53:04.121840 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:04.121794 2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="opendatahub/maas-api-6b989f9fdb-nfrzb" podStartSLOduration=3.094987582 podStartE2EDuration="5.12177687s" podCreationTimestamp="2026-06-02 01:52:59 +0000 UTC" firstStartedPulling="2026-06-02 01:53:01.625270235 +0000 UTC m=+699.271076855" lastFinishedPulling="2026-06-02 01:53:03.652059515 +0000 UTC m=+701.297866143" observedRunningTime="2026-06-02 01:53:04.119336858 +0000 UTC m=+701.765143501" watchObservedRunningTime="2026-06-02 01:53:04.12177687 +0000 UTC m=+701.767583511" Jun 02 01:53:10.114860 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:10.114778 2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-6b989f9fdb-nfrzb" Jun 02 01:53:11.860966 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:11.860928 2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["keycloak-system/maas-keycloak-0"] Jun 02 01:53:11.861429 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:11.861179 2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="keycloak-system/maas-keycloak-0" podUID="b84a29f8-554d-4eb2-ab8e-0608a11f3ad5" containerName="keycloak" containerID="cri-o://7014313f6ac5b5fb1596709e0809ea14ea2200d02818156f0eee7864574f1040" gracePeriod=30 Jun 02 01:53:12.419063 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:12.419014 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="keycloak-system/maas-keycloak-0" podUID="b84a29f8-554d-4eb2-ab8e-0608a11f3ad5" containerName="keycloak" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jun 02 01:53:13.902584 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:13.902559 2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="keycloak-system/maas-keycloak-0" Jun 02 01:53:13.968823 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:13.968788 2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xmz7h\" (UniqueName: \"kubernetes.io/projected/b84a29f8-554d-4eb2-ab8e-0608a11f3ad5-kube-api-access-xmz7h\") pod \"b84a29f8-554d-4eb2-ab8e-0608a11f3ad5\" (UID: \"b84a29f8-554d-4eb2-ab8e-0608a11f3ad5\") " Jun 02 01:53:13.971157 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:13.971116 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b84a29f8-554d-4eb2-ab8e-0608a11f3ad5-kube-api-access-xmz7h" (OuterVolumeSpecName: "kube-api-access-xmz7h") pod "b84a29f8-554d-4eb2-ab8e-0608a11f3ad5" (UID: "b84a29f8-554d-4eb2-ab8e-0608a11f3ad5"). InnerVolumeSpecName "kube-api-access-xmz7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 02 01:53:14.069967 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:14.069874 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmz7h\" (UniqueName: \"kubernetes.io/projected/b84a29f8-554d-4eb2-ab8e-0608a11f3ad5-kube-api-access-xmz7h\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\"" Jun 02 01:53:14.142498 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:14.142460 2571 generic.go:358] "Generic (PLEG): container finished" podID="b84a29f8-554d-4eb2-ab8e-0608a11f3ad5" containerID="7014313f6ac5b5fb1596709e0809ea14ea2200d02818156f0eee7864574f1040" exitCode=143 Jun 02 01:53:14.142668 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:14.142507 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="keycloak-system/maas-keycloak-0" event={"ID":"b84a29f8-554d-4eb2-ab8e-0608a11f3ad5","Type":"ContainerDied","Data":"7014313f6ac5b5fb1596709e0809ea14ea2200d02818156f0eee7864574f1040"} Jun 02 01:53:14.142668 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:14.142524 2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="keycloak-system/maas-keycloak-0" Jun 02 01:53:14.142668 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:14.142538 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="keycloak-system/maas-keycloak-0" event={"ID":"b84a29f8-554d-4eb2-ab8e-0608a11f3ad5","Type":"ContainerDied","Data":"df6a838dadf2c0bdf79dd666150e240e1f9c29fe2bd59d58b73149383b7e116b"} Jun 02 01:53:14.142668 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:14.142557 2571 scope.go:117] "RemoveContainer" containerID="7014313f6ac5b5fb1596709e0809ea14ea2200d02818156f0eee7864574f1040" Jun 02 01:53:14.152276 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:14.152258 2571 scope.go:117] "RemoveContainer" containerID="7014313f6ac5b5fb1596709e0809ea14ea2200d02818156f0eee7864574f1040" Jun 02 01:53:14.152532 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:53:14.152514 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7014313f6ac5b5fb1596709e0809ea14ea2200d02818156f0eee7864574f1040\": container with ID starting with 7014313f6ac5b5fb1596709e0809ea14ea2200d02818156f0eee7864574f1040 not found: ID does not exist" containerID="7014313f6ac5b5fb1596709e0809ea14ea2200d02818156f0eee7864574f1040" Jun 02 01:53:14.152592 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:14.152545 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7014313f6ac5b5fb1596709e0809ea14ea2200d02818156f0eee7864574f1040"} err="failed to get container status \"7014313f6ac5b5fb1596709e0809ea14ea2200d02818156f0eee7864574f1040\": rpc error: code = NotFound desc = could not find container \"7014313f6ac5b5fb1596709e0809ea14ea2200d02818156f0eee7864574f1040\": container with ID starting with 7014313f6ac5b5fb1596709e0809ea14ea2200d02818156f0eee7864574f1040 not found: ID does not exist" Jun 02 01:53:14.164318 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:14.164282 2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["keycloak-system/maas-keycloak-0"] Jun 02 01:53:14.167875 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:14.167850 2571 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["keycloak-system/maas-keycloak-0"] Jun 02 01:53:14.188648 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:14.188626 2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["keycloak-system/maas-keycloak-0"] Jun 02 01:53:14.188949 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:14.188937 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b84a29f8-554d-4eb2-ab8e-0608a11f3ad5" containerName="keycloak" Jun 02 01:53:14.188990 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:14.188952 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="b84a29f8-554d-4eb2-ab8e-0608a11f3ad5" containerName="keycloak" Jun 02 01:53:14.189030 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:14.189020 2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="b84a29f8-554d-4eb2-ab8e-0608a11f3ad5" containerName="keycloak" Jun 02 01:53:14.193505 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:14.193489 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="keycloak-system/maas-keycloak-0" Jun 02 01:53:14.196185 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:14.196164 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"keycloak-system\"/\"default-dockercfg-8d6dj\"" Jun 02 01:53:14.196185 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:14.196175 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"keycloak-system\"/\"openshift-service-ca.crt\"" Jun 02 01:53:14.196395 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:14.196171 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"keycloak-system\"/\"keycloak-test-realms\"" Jun 02 01:53:14.196395 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:14.196167 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"keycloak-system\"/\"maas-keycloak-initial-admin\"" Jun 02 01:53:14.196395 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:14.196171 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"keycloak-system\"/\"kube-root-ca.crt\"" Jun 02 01:53:14.201415 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:14.201396 2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["keycloak-system/maas-keycloak-0"] Jun 02 01:53:14.271882 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:14.271850 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-realms\" (UniqueName: \"kubernetes.io/configmap/1d72a536-cf79-4308-809d-506a61a03ca4-test-realms\") pod \"maas-keycloak-0\" (UID: \"1d72a536-cf79-4308-809d-506a61a03ca4\") " pod="keycloak-system/maas-keycloak-0" Jun 02 01:53:14.272056 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:14.271935 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcq4f\" (UniqueName: \"kubernetes.io/projected/1d72a536-cf79-4308-809d-506a61a03ca4-kube-api-access-gcq4f\") pod \"maas-keycloak-0\" (UID: \"1d72a536-cf79-4308-809d-506a61a03ca4\") " pod="keycloak-system/maas-keycloak-0" Jun 02 01:53:14.372905 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:14.372802 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gcq4f\" (UniqueName: \"kubernetes.io/projected/1d72a536-cf79-4308-809d-506a61a03ca4-kube-api-access-gcq4f\") pod \"maas-keycloak-0\" (UID: \"1d72a536-cf79-4308-809d-506a61a03ca4\") " pod="keycloak-system/maas-keycloak-0" Jun 02 01:53:14.373091 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:14.372976 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"test-realms\" (UniqueName: \"kubernetes.io/configmap/1d72a536-cf79-4308-809d-506a61a03ca4-test-realms\") pod \"maas-keycloak-0\" (UID: \"1d72a536-cf79-4308-809d-506a61a03ca4\") " pod="keycloak-system/maas-keycloak-0" Jun 02 01:53:14.373638 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:14.373620 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-realms\" (UniqueName: \"kubernetes.io/configmap/1d72a536-cf79-4308-809d-506a61a03ca4-test-realms\") pod \"maas-keycloak-0\" (UID: \"1d72a536-cf79-4308-809d-506a61a03ca4\") " pod="keycloak-system/maas-keycloak-0" Jun 02 01:53:14.381065 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:14.381041 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcq4f\" (UniqueName: \"kubernetes.io/projected/1d72a536-cf79-4308-809d-506a61a03ca4-kube-api-access-gcq4f\") pod \"maas-keycloak-0\" (UID: \"1d72a536-cf79-4308-809d-506a61a03ca4\") " pod="keycloak-system/maas-keycloak-0" Jun 02 01:53:14.503553 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:14.503519 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="keycloak-system/maas-keycloak-0" Jun 02 01:53:14.629784 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:14.629760 2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["keycloak-system/maas-keycloak-0"] Jun 02 01:53:14.631846 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:53:14.631812 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d72a536_cf79_4308_809d_506a61a03ca4.slice/crio-e1fec19897d57a013bc5fc93edfc12c74ea9553532b986937ccba9442c178a23 WatchSource:0}: Error finding container e1fec19897d57a013bc5fc93edfc12c74ea9553532b986937ccba9442c178a23: Status 404 returned error can't find the container with id e1fec19897d57a013bc5fc93edfc12c74ea9553532b986937ccba9442c178a23 Jun 02 01:53:14.928534 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:14.928497 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b84a29f8-554d-4eb2-ab8e-0608a11f3ad5" path="/var/lib/kubelet/pods/b84a29f8-554d-4eb2-ab8e-0608a11f3ad5/volumes" Jun 02 01:53:15.149600 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:15.149557 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="keycloak-system/maas-keycloak-0" event={"ID":"1d72a536-cf79-4308-809d-506a61a03ca4","Type":"ContainerStarted","Data":"23136b90231263810e7a953970fc5bcd70219b9171b7ffaba9e50d3d99e66749"} Jun 02 01:53:15.149600 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:15.149599 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="keycloak-system/maas-keycloak-0" event={"ID":"1d72a536-cf79-4308-809d-506a61a03ca4","Type":"ContainerStarted","Data":"e1fec19897d57a013bc5fc93edfc12c74ea9553532b986937ccba9442c178a23"} Jun 02 01:53:15.168369 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:15.168311 2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="keycloak-system/maas-keycloak-0" podStartSLOduration=0.867619019 podStartE2EDuration="1.168297052s" podCreationTimestamp="2026-06-02 01:53:14 +0000 UTC" firstStartedPulling="2026-06-02 01:53:14.63323022 +0000 UTC m=+712.279036840" lastFinishedPulling="2026-06-02 01:53:14.933908249 +0000 UTC m=+712.579714873" observedRunningTime="2026-06-02 01:53:15.166572865 +0000 UTC m=+712.812379507" watchObservedRunningTime="2026-06-02 01:53:15.168297052 +0000 UTC m=+712.814103694" Jun 02 01:53:15.504147 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:15.504106 2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="keycloak-system/maas-keycloak-0" Jun 02 01:53:15.505263 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:15.505232 2571 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="1d72a536-cf79-4308-809d-506a61a03ca4" containerName="keycloak" probeResult="failure" output="Get \"http://10.134.0.31:9000/health/started\": dial tcp 10.134.0.31:9000: connect: connection refused" Jun 02 01:53:16.504822 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:16.504771 2571 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="1d72a536-cf79-4308-809d-506a61a03ca4" containerName="keycloak" probeResult="failure" output="Get \"http://10.134.0.31:9000/health/started\": dial tcp 10.134.0.31:9000: connect: connection refused" Jun 02 01:53:17.504145 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:17.504082 2571 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="1d72a536-cf79-4308-809d-506a61a03ca4" containerName="keycloak" probeResult="failure" output="Get \"http://10.134.0.31:9000/health/started\": dial tcp 10.134.0.31:9000: connect: connection refused" Jun 02 01:53:18.504926 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:18.504879 2571 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="1d72a536-cf79-4308-809d-506a61a03ca4" containerName="keycloak" probeResult="failure" output="Get \"http://10.134.0.31:9000/health/started\": dial tcp 10.134.0.31:9000: connect: connection refused" Jun 02 01:53:19.504683 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:19.504636 2571 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="1d72a536-cf79-4308-809d-506a61a03ca4" containerName="keycloak" probeResult="failure" output="Get \"http://10.134.0.31:9000/health/started\": dial tcp 10.134.0.31:9000: connect: connection refused" Jun 02 01:53:20.505000 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:20.504928 2571 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="1d72a536-cf79-4308-809d-506a61a03ca4" containerName="keycloak" probeResult="failure" output="Get \"http://10.134.0.31:9000/health/started\": dial tcp 10.134.0.31:9000: connect: connection refused" Jun 02 01:53:21.504082 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:21.504031 2571 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="1d72a536-cf79-4308-809d-506a61a03ca4" containerName="keycloak" probeResult="failure" output="Get \"http://10.134.0.31:9000/health/started\": dial tcp 10.134.0.31:9000: connect: connection refused" Jun 02 01:53:22.504237 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:22.504182 2571 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="1d72a536-cf79-4308-809d-506a61a03ca4" containerName="keycloak" probeResult="failure" output="Get \"http://10.134.0.31:9000/health/started\": dial tcp 10.134.0.31:9000: connect: connection refused" Jun 02 01:53:23.504740 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:23.504689 2571 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="1d72a536-cf79-4308-809d-506a61a03ca4" containerName="keycloak" probeResult="failure" output="Get \"http://10.134.0.31:9000/health/started\": dial tcp 10.134.0.31:9000: connect: connection refused" Jun 02 01:53:24.503789 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:24.503752 2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="keycloak-system/maas-keycloak-0" Jun 02 01:53:24.504798 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:24.504763 2571 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="1d72a536-cf79-4308-809d-506a61a03ca4" containerName="keycloak" probeResult="failure" output="Get \"http://10.134.0.31:9000/health/started\": dial tcp 10.134.0.31:9000: connect: connection refused" Jun 02 01:53:25.504221 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:25.504172 2571 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="1d72a536-cf79-4308-809d-506a61a03ca4" containerName="keycloak" probeResult="failure" output="Get \"http://10.134.0.31:9000/health/started\": dial tcp 10.134.0.31:9000: connect: connection refused" Jun 02 01:53:26.504780 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:26.504732 2571 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="1d72a536-cf79-4308-809d-506a61a03ca4" containerName="keycloak" probeResult="failure" output="Get \"http://10.134.0.31:9000/health/started\": dial tcp 10.134.0.31:9000: connect: connection refused" Jun 02 01:53:27.504214 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:27.504162 2571 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="1d72a536-cf79-4308-809d-506a61a03ca4" containerName="keycloak" probeResult="failure" output="Get \"http://10.134.0.31:9000/health/started\": dial tcp 10.134.0.31:9000: connect: connection refused" Jun 02 01:53:28.636187 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:28.636142 2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="keycloak-system/maas-keycloak-0" Jun 02 01:53:28.653364 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:28.653316 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="keycloak-system/maas-keycloak-0" podUID="1d72a536-cf79-4308-809d-506a61a03ca4" containerName="keycloak" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jun 02 01:53:38.643056 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:38.643016 2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="keycloak-system/maas-keycloak-0" Jun 02 01:53:49.327285 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:49.327248 2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-544d5c477f-xhxr8"] Jun 02 01:53:49.330793 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:49.330771 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-544d5c477f-xhxr8" Jun 02 01:53:49.333386 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:49.333365 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"authorino-oidc-ca\"" Jun 02 01:53:49.338902 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:49.338610 2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-544d5c477f-xhxr8"] Jun 02 01:53:49.434078 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:49.434041 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/08e0b510-bb35-47e4-beac-dac9dc53adf7-tls-cert\") pod \"authorino-544d5c477f-xhxr8\" (UID: \"08e0b510-bb35-47e4-beac-dac9dc53adf7\") " pod="kuadrant-system/authorino-544d5c477f-xhxr8" Jun 02 01:53:49.434078 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:49.434078 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oidc-ca\" (UniqueName: \"kubernetes.io/configmap/08e0b510-bb35-47e4-beac-dac9dc53adf7-oidc-ca\") pod \"authorino-544d5c477f-xhxr8\" (UID: \"08e0b510-bb35-47e4-beac-dac9dc53adf7\") " pod="kuadrant-system/authorino-544d5c477f-xhxr8" Jun 02 01:53:49.434366 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:49.434118 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79dwk\" (UniqueName: \"kubernetes.io/projected/08e0b510-bb35-47e4-beac-dac9dc53adf7-kube-api-access-79dwk\") pod \"authorino-544d5c477f-xhxr8\" (UID: \"08e0b510-bb35-47e4-beac-dac9dc53adf7\") " pod="kuadrant-system/authorino-544d5c477f-xhxr8" Jun 02 01:53:49.535040 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:49.534992 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/08e0b510-bb35-47e4-beac-dac9dc53adf7-tls-cert\") pod \"authorino-544d5c477f-xhxr8\" (UID: \"08e0b510-bb35-47e4-beac-dac9dc53adf7\") " pod="kuadrant-system/authorino-544d5c477f-xhxr8" Jun 02 01:53:49.535040 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:49.535039 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oidc-ca\" (UniqueName: \"kubernetes.io/configmap/08e0b510-bb35-47e4-beac-dac9dc53adf7-oidc-ca\") pod \"authorino-544d5c477f-xhxr8\" (UID: \"08e0b510-bb35-47e4-beac-dac9dc53adf7\") " pod="kuadrant-system/authorino-544d5c477f-xhxr8" Jun 02 01:53:49.535277 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:49.535075 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-79dwk\" (UniqueName: \"kubernetes.io/projected/08e0b510-bb35-47e4-beac-dac9dc53adf7-kube-api-access-79dwk\") pod \"authorino-544d5c477f-xhxr8\" (UID: \"08e0b510-bb35-47e4-beac-dac9dc53adf7\") " pod="kuadrant-system/authorino-544d5c477f-xhxr8" Jun 02 01:53:49.535835 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:49.535807 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oidc-ca\" (UniqueName: \"kubernetes.io/configmap/08e0b510-bb35-47e4-beac-dac9dc53adf7-oidc-ca\") pod \"authorino-544d5c477f-xhxr8\" (UID: \"08e0b510-bb35-47e4-beac-dac9dc53adf7\") " pod="kuadrant-system/authorino-544d5c477f-xhxr8" Jun 02 01:53:49.537327 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:49.537312 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/08e0b510-bb35-47e4-beac-dac9dc53adf7-tls-cert\") pod \"authorino-544d5c477f-xhxr8\" (UID: \"08e0b510-bb35-47e4-beac-dac9dc53adf7\") " pod="kuadrant-system/authorino-544d5c477f-xhxr8" Jun 02 01:53:49.543912 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:49.543884 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79dwk\" (UniqueName: \"kubernetes.io/projected/08e0b510-bb35-47e4-beac-dac9dc53adf7-kube-api-access-79dwk\") pod \"authorino-544d5c477f-xhxr8\" (UID: \"08e0b510-bb35-47e4-beac-dac9dc53adf7\") " pod="kuadrant-system/authorino-544d5c477f-xhxr8" Jun 02 01:53:49.641227 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:49.641200 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-544d5c477f-xhxr8" Jun 02 01:53:49.765265 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:49.765204 2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-544d5c477f-xhxr8"] Jun 02 01:53:49.768402 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:53:49.768369 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod08e0b510_bb35_47e4_beac_dac9dc53adf7.slice/crio-b5f5dc092c74408f604850e2a91b6bfa99f7cb4475ab841294b34f3c31f1d08f WatchSource:0}: Error finding container b5f5dc092c74408f604850e2a91b6bfa99f7cb4475ab841294b34f3c31f1d08f: Status 404 returned error can't find the container with id b5f5dc092c74408f604850e2a91b6bfa99f7cb4475ab841294b34f3c31f1d08f Jun 02 01:53:50.282153 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:50.282108 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-544d5c477f-xhxr8" event={"ID":"08e0b510-bb35-47e4-beac-dac9dc53adf7","Type":"ContainerStarted","Data":"6313b959850cea10f660b6186c6dcec6b6bf6e3b255acd527a2d53b2df0ab579"} Jun 02 01:53:50.282306 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:50.282161 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-544d5c477f-xhxr8" event={"ID":"08e0b510-bb35-47e4-beac-dac9dc53adf7","Type":"ContainerStarted","Data":"b5f5dc092c74408f604850e2a91b6bfa99f7cb4475ab841294b34f3c31f1d08f"} Jun 02 01:53:50.298520 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:50.298470 2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/authorino-544d5c477f-xhxr8" podStartSLOduration=0.919936537 podStartE2EDuration="1.298456592s" podCreationTimestamp="2026-06-02 01:53:49 +0000 UTC" firstStartedPulling="2026-06-02 01:53:49.769703758 +0000 UTC m=+747.415510378" lastFinishedPulling="2026-06-02 01:53:50.14822381 +0000 UTC m=+747.794030433" observedRunningTime="2026-06-02 01:53:50.296555595 +0000 UTC m=+747.942362238" watchObservedRunningTime="2026-06-02 01:53:50.298456592 +0000 UTC m=+747.944263289" Jun 02 01:53:50.323904 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:50.323867 2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-586bfc7ddf-ccmzj"] Jun 02 01:53:50.324166 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:50.324120 2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="kuadrant-system/authorino-586bfc7ddf-ccmzj" podUID="9962c95a-a54c-4096-9d7a-85a9e6b777e7" containerName="authorino" containerID="cri-o://c5ab2a058f7589aeffd134e7f738ed0dcf90719df6a96e8f20075dde80a5e46d" gracePeriod=30 Jun 02 01:53:50.583472 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:50.583446 2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-586bfc7ddf-ccmzj" Jun 02 01:53:50.644527 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:50.644485 2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v6fj8\" (UniqueName: \"kubernetes.io/projected/9962c95a-a54c-4096-9d7a-85a9e6b777e7-kube-api-access-v6fj8\") pod \"9962c95a-a54c-4096-9d7a-85a9e6b777e7\" (UID: \"9962c95a-a54c-4096-9d7a-85a9e6b777e7\") " Jun 02 01:53:50.644700 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:50.644550 2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9962c95a-a54c-4096-9d7a-85a9e6b777e7-tls-cert\") pod \"9962c95a-a54c-4096-9d7a-85a9e6b777e7\" (UID: \"9962c95a-a54c-4096-9d7a-85a9e6b777e7\") " Jun 02 01:53:50.646612 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:50.646582 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9962c95a-a54c-4096-9d7a-85a9e6b777e7-kube-api-access-v6fj8" (OuterVolumeSpecName: "kube-api-access-v6fj8") pod "9962c95a-a54c-4096-9d7a-85a9e6b777e7" (UID: "9962c95a-a54c-4096-9d7a-85a9e6b777e7"). InnerVolumeSpecName "kube-api-access-v6fj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 02 01:53:50.655724 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:50.655697 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9962c95a-a54c-4096-9d7a-85a9e6b777e7-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "9962c95a-a54c-4096-9d7a-85a9e6b777e7" (UID: "9962c95a-a54c-4096-9d7a-85a9e6b777e7"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 02 01:53:50.746226 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:50.746147 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6fj8\" (UniqueName: \"kubernetes.io/projected/9962c95a-a54c-4096-9d7a-85a9e6b777e7-kube-api-access-v6fj8\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\"" Jun 02 01:53:50.746226 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:50.746178 2571 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9962c95a-a54c-4096-9d7a-85a9e6b777e7-tls-cert\") on node \"ip-10-0-135-117.ec2.internal\" DevicePath \"\"" Jun 02 01:53:51.286320 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:51.286280 2571 generic.go:358] "Generic (PLEG): container finished" podID="9962c95a-a54c-4096-9d7a-85a9e6b777e7" containerID="c5ab2a058f7589aeffd134e7f738ed0dcf90719df6a96e8f20075dde80a5e46d" exitCode=0 Jun 02 01:53:51.286514 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:51.286350 2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-586bfc7ddf-ccmzj" Jun 02 01:53:51.286514 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:51.286369 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-586bfc7ddf-ccmzj" event={"ID":"9962c95a-a54c-4096-9d7a-85a9e6b777e7","Type":"ContainerDied","Data":"c5ab2a058f7589aeffd134e7f738ed0dcf90719df6a96e8f20075dde80a5e46d"} Jun 02 01:53:51.286514 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:51.286414 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-586bfc7ddf-ccmzj" event={"ID":"9962c95a-a54c-4096-9d7a-85a9e6b777e7","Type":"ContainerDied","Data":"83a521d44c73d4e7eb9ba985e5b702d8e5c15981d3048e0d238c1fad76f8b6f9"} Jun 02 01:53:51.286514 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:51.286432 2571 scope.go:117] "RemoveContainer" containerID="c5ab2a058f7589aeffd134e7f738ed0dcf90719df6a96e8f20075dde80a5e46d" Jun 02 01:53:51.294235 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:51.294213 2571 scope.go:117] "RemoveContainer" containerID="c5ab2a058f7589aeffd134e7f738ed0dcf90719df6a96e8f20075dde80a5e46d" Jun 02 01:53:51.294484 ip-10-0-135-117 kubenswrapper[2571]: E0602 01:53:51.294461 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5ab2a058f7589aeffd134e7f738ed0dcf90719df6a96e8f20075dde80a5e46d\": container with ID starting with c5ab2a058f7589aeffd134e7f738ed0dcf90719df6a96e8f20075dde80a5e46d not found: ID does not exist" containerID="c5ab2a058f7589aeffd134e7f738ed0dcf90719df6a96e8f20075dde80a5e46d" Jun 02 01:53:51.294563 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:51.294491 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5ab2a058f7589aeffd134e7f738ed0dcf90719df6a96e8f20075dde80a5e46d"} err="failed to get container status \"c5ab2a058f7589aeffd134e7f738ed0dcf90719df6a96e8f20075dde80a5e46d\": rpc error: code = NotFound desc = could not find container \"c5ab2a058f7589aeffd134e7f738ed0dcf90719df6a96e8f20075dde80a5e46d\": container with ID starting with c5ab2a058f7589aeffd134e7f738ed0dcf90719df6a96e8f20075dde80a5e46d not found: ID does not exist" Jun 02 01:53:51.302766 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:51.302742 2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-586bfc7ddf-ccmzj"] Jun 02 01:53:51.308518 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:51.308496 2571 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-586bfc7ddf-ccmzj"] Jun 02 01:53:52.928650 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:53:52.928617 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9962c95a-a54c-4096-9d7a-85a9e6b777e7" path="/var/lib/kubelet/pods/9962c95a-a54c-4096-9d7a-85a9e6b777e7/volumes" Jun 02 01:54:14.980102 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:14.980068 2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-5489f4f4bc-wh9lm"] Jun 02 01:54:14.980585 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:14.980425 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9962c95a-a54c-4096-9d7a-85a9e6b777e7" containerName="authorino" Jun 02 01:54:14.980585 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:14.980438 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9962c95a-a54c-4096-9d7a-85a9e6b777e7" containerName="authorino" Jun 02 01:54:14.980585 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:14.980492 2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="9962c95a-a54c-4096-9d7a-85a9e6b777e7" containerName="authorino" Jun 02 01:54:14.983769 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:14.983744 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-5489f4f4bc-wh9lm" Jun 02 01:54:14.987590 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:14.987562 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-4c4k9\"" Jun 02 01:54:14.987731 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:14.987612 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jun 02 01:54:14.987731 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:14.987635 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jun 02 01:54:14.987731 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:14.987714 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jun 02 01:54:14.994222 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:14.994199 2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-5489f4f4bc-wh9lm"] Jun 02 01:54:15.057061 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:15.057031 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88e4f184-ec91-493d-9fc3-b09fee7a72ce-tls-certs\") pod \"e2e-distinct-simulated-kserve-5489f4f4bc-wh9lm\" (UID: \"88e4f184-ec91-493d-9fc3-b09fee7a72ce\") " pod="llm/e2e-distinct-simulated-kserve-5489f4f4bc-wh9lm" Jun 02 01:54:15.057061 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:15.057065 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8km2b\" (UniqueName: \"kubernetes.io/projected/88e4f184-ec91-493d-9fc3-b09fee7a72ce-kube-api-access-8km2b\") pod \"e2e-distinct-simulated-kserve-5489f4f4bc-wh9lm\" (UID: \"88e4f184-ec91-493d-9fc3-b09fee7a72ce\") " pod="llm/e2e-distinct-simulated-kserve-5489f4f4bc-wh9lm" Jun 02 01:54:15.057275 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:15.057086 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88e4f184-ec91-493d-9fc3-b09fee7a72ce-kserve-provision-location\") pod \"e2e-distinct-simulated-kserve-5489f4f4bc-wh9lm\" (UID: \"88e4f184-ec91-493d-9fc3-b09fee7a72ce\") " pod="llm/e2e-distinct-simulated-kserve-5489f4f4bc-wh9lm" Jun 02 01:54:15.057275 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:15.057185 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88e4f184-ec91-493d-9fc3-b09fee7a72ce-home\") pod \"e2e-distinct-simulated-kserve-5489f4f4bc-wh9lm\" (UID: \"88e4f184-ec91-493d-9fc3-b09fee7a72ce\") " pod="llm/e2e-distinct-simulated-kserve-5489f4f4bc-wh9lm" Jun 02 01:54:15.057275 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:15.057210 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88e4f184-ec91-493d-9fc3-b09fee7a72ce-dshm\") pod \"e2e-distinct-simulated-kserve-5489f4f4bc-wh9lm\" (UID: \"88e4f184-ec91-493d-9fc3-b09fee7a72ce\") " pod="llm/e2e-distinct-simulated-kserve-5489f4f4bc-wh9lm" Jun 02 01:54:15.057374 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:15.057276 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88e4f184-ec91-493d-9fc3-b09fee7a72ce-model-cache\") pod \"e2e-distinct-simulated-kserve-5489f4f4bc-wh9lm\" (UID: \"88e4f184-ec91-493d-9fc3-b09fee7a72ce\") " pod="llm/e2e-distinct-simulated-kserve-5489f4f4bc-wh9lm" Jun 02 01:54:15.158195 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:15.158154 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88e4f184-ec91-493d-9fc3-b09fee7a72ce-model-cache\") pod \"e2e-distinct-simulated-kserve-5489f4f4bc-wh9lm\" (UID: \"88e4f184-ec91-493d-9fc3-b09fee7a72ce\") " pod="llm/e2e-distinct-simulated-kserve-5489f4f4bc-wh9lm" Jun 02 01:54:15.158384 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:15.158225 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88e4f184-ec91-493d-9fc3-b09fee7a72ce-tls-certs\") pod \"e2e-distinct-simulated-kserve-5489f4f4bc-wh9lm\" (UID: \"88e4f184-ec91-493d-9fc3-b09fee7a72ce\") " pod="llm/e2e-distinct-simulated-kserve-5489f4f4bc-wh9lm" Jun 02 01:54:15.158384 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:15.158244 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8km2b\" (UniqueName: \"kubernetes.io/projected/88e4f184-ec91-493d-9fc3-b09fee7a72ce-kube-api-access-8km2b\") pod \"e2e-distinct-simulated-kserve-5489f4f4bc-wh9lm\" (UID: \"88e4f184-ec91-493d-9fc3-b09fee7a72ce\") " pod="llm/e2e-distinct-simulated-kserve-5489f4f4bc-wh9lm" Jun 02 01:54:15.158384 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:15.158262 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88e4f184-ec91-493d-9fc3-b09fee7a72ce-kserve-provision-location\") pod \"e2e-distinct-simulated-kserve-5489f4f4bc-wh9lm\" (UID: \"88e4f184-ec91-493d-9fc3-b09fee7a72ce\") " pod="llm/e2e-distinct-simulated-kserve-5489f4f4bc-wh9lm" Jun 02 01:54:15.158384 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:15.158291 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88e4f184-ec91-493d-9fc3-b09fee7a72ce-home\") pod \"e2e-distinct-simulated-kserve-5489f4f4bc-wh9lm\" (UID: \"88e4f184-ec91-493d-9fc3-b09fee7a72ce\") " pod="llm/e2e-distinct-simulated-kserve-5489f4f4bc-wh9lm" Jun 02 01:54:15.158384 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:15.158306 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88e4f184-ec91-493d-9fc3-b09fee7a72ce-dshm\") pod \"e2e-distinct-simulated-kserve-5489f4f4bc-wh9lm\" (UID: \"88e4f184-ec91-493d-9fc3-b09fee7a72ce\") " pod="llm/e2e-distinct-simulated-kserve-5489f4f4bc-wh9lm" Jun 02 01:54:15.158657 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:15.158633 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88e4f184-ec91-493d-9fc3-b09fee7a72ce-model-cache\") pod \"e2e-distinct-simulated-kserve-5489f4f4bc-wh9lm\" (UID: \"88e4f184-ec91-493d-9fc3-b09fee7a72ce\") " pod="llm/e2e-distinct-simulated-kserve-5489f4f4bc-wh9lm" Jun 02 01:54:15.158702 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:15.158665 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88e4f184-ec91-493d-9fc3-b09fee7a72ce-kserve-provision-location\") pod \"e2e-distinct-simulated-kserve-5489f4f4bc-wh9lm\" (UID: \"88e4f184-ec91-493d-9fc3-b09fee7a72ce\") " pod="llm/e2e-distinct-simulated-kserve-5489f4f4bc-wh9lm" Jun 02 01:54:15.158741 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:15.158698 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88e4f184-ec91-493d-9fc3-b09fee7a72ce-home\") pod \"e2e-distinct-simulated-kserve-5489f4f4bc-wh9lm\" (UID: \"88e4f184-ec91-493d-9fc3-b09fee7a72ce\") " pod="llm/e2e-distinct-simulated-kserve-5489f4f4bc-wh9lm" Jun 02 01:54:15.160552 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:15.160526 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88e4f184-ec91-493d-9fc3-b09fee7a72ce-dshm\") pod \"e2e-distinct-simulated-kserve-5489f4f4bc-wh9lm\" (UID: \"88e4f184-ec91-493d-9fc3-b09fee7a72ce\") " pod="llm/e2e-distinct-simulated-kserve-5489f4f4bc-wh9lm" Jun 02 01:54:15.160803 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:15.160787 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88e4f184-ec91-493d-9fc3-b09fee7a72ce-tls-certs\") pod \"e2e-distinct-simulated-kserve-5489f4f4bc-wh9lm\" (UID: \"88e4f184-ec91-493d-9fc3-b09fee7a72ce\") " pod="llm/e2e-distinct-simulated-kserve-5489f4f4bc-wh9lm" Jun 02 01:54:15.166428 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:15.166398 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8km2b\" (UniqueName: \"kubernetes.io/projected/88e4f184-ec91-493d-9fc3-b09fee7a72ce-kube-api-access-8km2b\") pod \"e2e-distinct-simulated-kserve-5489f4f4bc-wh9lm\" (UID: \"88e4f184-ec91-493d-9fc3-b09fee7a72ce\") " pod="llm/e2e-distinct-simulated-kserve-5489f4f4bc-wh9lm" Jun 02 01:54:15.297928 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:15.297844 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-5489f4f4bc-wh9lm" Jun 02 01:54:15.426972 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:15.426941 2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-5489f4f4bc-wh9lm"] Jun 02 01:54:15.429688 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:54:15.429662 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88e4f184_ec91_493d_9fc3_b09fee7a72ce.slice/crio-52304be46183e0927d31c8438bd71c7ed6021d5b3a81bb9bbb3d0e7ff9628e11 WatchSource:0}: Error finding container 52304be46183e0927d31c8438bd71c7ed6021d5b3a81bb9bbb3d0e7ff9628e11: Status 404 returned error can't find the container with id 52304be46183e0927d31c8438bd71c7ed6021d5b3a81bb9bbb3d0e7ff9628e11 Jun 02 01:54:16.364657 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:16.364622 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-5489f4f4bc-wh9lm" event={"ID":"88e4f184-ec91-493d-9fc3-b09fee7a72ce","Type":"ContainerStarted","Data":"52304be46183e0927d31c8438bd71c7ed6021d5b3a81bb9bbb3d0e7ff9628e11"} Jun 02 01:54:19.376794 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:19.376748 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-5489f4f4bc-wh9lm" event={"ID":"88e4f184-ec91-493d-9fc3-b09fee7a72ce","Type":"ContainerStarted","Data":"6f32223132d695068c81eed83d89b51ca9d7a0a174567bf965e67013d15a1866"} Jun 02 01:54:20.381493 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:20.381450 2571 generic.go:358] "Generic (PLEG): container finished" podID="88e4f184-ec91-493d-9fc3-b09fee7a72ce" containerID="6f32223132d695068c81eed83d89b51ca9d7a0a174567bf965e67013d15a1866" exitCode=0 Jun 02 01:54:20.381946 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:20.381536 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-5489f4f4bc-wh9lm" event={"ID":"88e4f184-ec91-493d-9fc3-b09fee7a72ce","Type":"ContainerDied","Data":"6f32223132d695068c81eed83d89b51ca9d7a0a174567bf965e67013d15a1866"} Jun 02 01:54:22.390814 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:22.390780 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-5489f4f4bc-wh9lm" event={"ID":"88e4f184-ec91-493d-9fc3-b09fee7a72ce","Type":"ContainerStarted","Data":"f12abdaaa71de966d955de8f1082db08ebfe5d300d21b1f337dd9676ef15fc68"} Jun 02 01:54:22.411629 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:22.411581 2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-5489f4f4bc-wh9lm" podStartSLOduration=2.454941643 podStartE2EDuration="8.411566445s" podCreationTimestamp="2026-06-02 01:54:14 +0000 UTC" firstStartedPulling="2026-06-02 01:54:15.431428926 +0000 UTC m=+773.077235546" lastFinishedPulling="2026-06-02 01:54:21.388053711 +0000 UTC m=+779.033860348" observedRunningTime="2026-06-02 01:54:22.409106256 +0000 UTC m=+780.054912897" watchObservedRunningTime="2026-06-02 01:54:22.411566445 +0000 UTC m=+780.057373087" Jun 02 01:54:25.298563 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:25.298519 2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-5489f4f4bc-wh9lm" Jun 02 01:54:25.298563 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:25.298565 2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-5489f4f4bc-wh9lm" Jun 02 01:54:25.311085 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:25.311060 2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-5489f4f4bc-wh9lm" Jun 02 01:54:25.410775 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:25.410745 2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-5489f4f4bc-wh9lm" Jun 02 01:54:36.277238 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:36.277203 2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-59dcdf78f5-7qx6c"] Jun 02 01:54:36.294194 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:36.294158 2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-59dcdf78f5-7qx6c"] Jun 02 01:54:36.294360 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:36.294296 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-59dcdf78f5-7qx6c" Jun 02 01:54:36.296981 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:36.296943 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jun 02 01:54:36.453508 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:36.453476 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a516b703-7511-4797-bcbe-763ca356f187-home\") pod \"facebook-opt-125m-simulated-kserve-59dcdf78f5-7qx6c\" (UID: \"a516b703-7511-4797-bcbe-763ca356f187\") " pod="llm/facebook-opt-125m-simulated-kserve-59dcdf78f5-7qx6c" Jun 02 01:54:36.453508 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:36.453516 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a516b703-7511-4797-bcbe-763ca356f187-kserve-provision-location\") pod \"facebook-opt-125m-simulated-kserve-59dcdf78f5-7qx6c\" (UID: \"a516b703-7511-4797-bcbe-763ca356f187\") " pod="llm/facebook-opt-125m-simulated-kserve-59dcdf78f5-7qx6c" Jun 02 01:54:36.453729 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:36.453624 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a516b703-7511-4797-bcbe-763ca356f187-model-cache\") pod \"facebook-opt-125m-simulated-kserve-59dcdf78f5-7qx6c\" (UID: \"a516b703-7511-4797-bcbe-763ca356f187\") " pod="llm/facebook-opt-125m-simulated-kserve-59dcdf78f5-7qx6c" Jun 02 01:54:36.453729 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:36.453698 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a516b703-7511-4797-bcbe-763ca356f187-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-59dcdf78f5-7qx6c\" (UID: \"a516b703-7511-4797-bcbe-763ca356f187\") " pod="llm/facebook-opt-125m-simulated-kserve-59dcdf78f5-7qx6c" Jun 02 01:54:36.453808 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:36.453729 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf6nq\" (UniqueName: \"kubernetes.io/projected/a516b703-7511-4797-bcbe-763ca356f187-kube-api-access-zf6nq\") pod \"facebook-opt-125m-simulated-kserve-59dcdf78f5-7qx6c\" (UID: \"a516b703-7511-4797-bcbe-763ca356f187\") " pod="llm/facebook-opt-125m-simulated-kserve-59dcdf78f5-7qx6c" Jun 02 01:54:36.453808 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:36.453754 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a516b703-7511-4797-bcbe-763ca356f187-dshm\") pod \"facebook-opt-125m-simulated-kserve-59dcdf78f5-7qx6c\" (UID: \"a516b703-7511-4797-bcbe-763ca356f187\") " pod="llm/facebook-opt-125m-simulated-kserve-59dcdf78f5-7qx6c" Jun 02 01:54:36.554598 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:36.554511 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a516b703-7511-4797-bcbe-763ca356f187-home\") pod \"facebook-opt-125m-simulated-kserve-59dcdf78f5-7qx6c\" (UID: \"a516b703-7511-4797-bcbe-763ca356f187\") " pod="llm/facebook-opt-125m-simulated-kserve-59dcdf78f5-7qx6c" Jun 02 01:54:36.554598 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:36.554561 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a516b703-7511-4797-bcbe-763ca356f187-kserve-provision-location\") pod \"facebook-opt-125m-simulated-kserve-59dcdf78f5-7qx6c\" (UID: \"a516b703-7511-4797-bcbe-763ca356f187\") " pod="llm/facebook-opt-125m-simulated-kserve-59dcdf78f5-7qx6c" Jun 02 01:54:36.554812 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:36.554619 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a516b703-7511-4797-bcbe-763ca356f187-model-cache\") pod \"facebook-opt-125m-simulated-kserve-59dcdf78f5-7qx6c\" (UID: \"a516b703-7511-4797-bcbe-763ca356f187\") " pod="llm/facebook-opt-125m-simulated-kserve-59dcdf78f5-7qx6c" Jun 02 01:54:36.554812 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:36.554644 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a516b703-7511-4797-bcbe-763ca356f187-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-59dcdf78f5-7qx6c\" (UID: \"a516b703-7511-4797-bcbe-763ca356f187\") " pod="llm/facebook-opt-125m-simulated-kserve-59dcdf78f5-7qx6c" Jun 02 01:54:36.554812 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:36.554668 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zf6nq\" (UniqueName: \"kubernetes.io/projected/a516b703-7511-4797-bcbe-763ca356f187-kube-api-access-zf6nq\") pod \"facebook-opt-125m-simulated-kserve-59dcdf78f5-7qx6c\" (UID: \"a516b703-7511-4797-bcbe-763ca356f187\") " pod="llm/facebook-opt-125m-simulated-kserve-59dcdf78f5-7qx6c" Jun 02 01:54:36.554812 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:36.554700 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a516b703-7511-4797-bcbe-763ca356f187-dshm\") pod \"facebook-opt-125m-simulated-kserve-59dcdf78f5-7qx6c\" (UID: \"a516b703-7511-4797-bcbe-763ca356f187\") " pod="llm/facebook-opt-125m-simulated-kserve-59dcdf78f5-7qx6c" Jun 02 01:54:36.555034 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:36.554901 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a516b703-7511-4797-bcbe-763ca356f187-home\") pod \"facebook-opt-125m-simulated-kserve-59dcdf78f5-7qx6c\" (UID: \"a516b703-7511-4797-bcbe-763ca356f187\") " pod="llm/facebook-opt-125m-simulated-kserve-59dcdf78f5-7qx6c" Jun 02 01:54:36.555034 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:36.555014 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a516b703-7511-4797-bcbe-763ca356f187-kserve-provision-location\") pod \"facebook-opt-125m-simulated-kserve-59dcdf78f5-7qx6c\" (UID: \"a516b703-7511-4797-bcbe-763ca356f187\") " pod="llm/facebook-opt-125m-simulated-kserve-59dcdf78f5-7qx6c" Jun 02 01:54:36.555152 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:36.555070 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a516b703-7511-4797-bcbe-763ca356f187-model-cache\") pod \"facebook-opt-125m-simulated-kserve-59dcdf78f5-7qx6c\" (UID: \"a516b703-7511-4797-bcbe-763ca356f187\") " pod="llm/facebook-opt-125m-simulated-kserve-59dcdf78f5-7qx6c" Jun 02 01:54:36.556948 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:36.556923 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a516b703-7511-4797-bcbe-763ca356f187-dshm\") pod \"facebook-opt-125m-simulated-kserve-59dcdf78f5-7qx6c\" (UID: \"a516b703-7511-4797-bcbe-763ca356f187\") " pod="llm/facebook-opt-125m-simulated-kserve-59dcdf78f5-7qx6c" Jun 02 01:54:36.557152 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:36.557117 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a516b703-7511-4797-bcbe-763ca356f187-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-59dcdf78f5-7qx6c\" (UID: \"a516b703-7511-4797-bcbe-763ca356f187\") " pod="llm/facebook-opt-125m-simulated-kserve-59dcdf78f5-7qx6c" Jun 02 01:54:36.563729 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:36.563705 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf6nq\" (UniqueName: \"kubernetes.io/projected/a516b703-7511-4797-bcbe-763ca356f187-kube-api-access-zf6nq\") pod \"facebook-opt-125m-simulated-kserve-59dcdf78f5-7qx6c\" (UID: \"a516b703-7511-4797-bcbe-763ca356f187\") " pod="llm/facebook-opt-125m-simulated-kserve-59dcdf78f5-7qx6c" Jun 02 01:54:36.606815 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:36.606770 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-59dcdf78f5-7qx6c" Jun 02 01:54:36.741017 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:36.740977 2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-59dcdf78f5-7qx6c"] Jun 02 01:54:36.741230 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:54:36.741206 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda516b703_7511_4797_bcbe_763ca356f187.slice/crio-0f891b2b2c574034577ac9adc1f5f7e4c193a2ae93d5f568246aa65a762befb3 WatchSource:0}: Error finding container 0f891b2b2c574034577ac9adc1f5f7e4c193a2ae93d5f568246aa65a762befb3: Status 404 returned error can't find the container with id 0f891b2b2c574034577ac9adc1f5f7e4c193a2ae93d5f568246aa65a762befb3 Jun 02 01:54:37.440121 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:37.440081 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-59dcdf78f5-7qx6c" event={"ID":"a516b703-7511-4797-bcbe-763ca356f187","Type":"ContainerStarted","Data":"f4b0bd5fafa269ded97db10939483dcdb1e9965d165abfaaab42b019235be00d"} Jun 02 01:54:37.440121 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:37.440121 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-59dcdf78f5-7qx6c" event={"ID":"a516b703-7511-4797-bcbe-763ca356f187","Type":"ContainerStarted","Data":"0f891b2b2c574034577ac9adc1f5f7e4c193a2ae93d5f568246aa65a762befb3"} Jun 02 01:54:38.444788 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:38.444751 2571 generic.go:358] "Generic (PLEG): container finished" podID="a516b703-7511-4797-bcbe-763ca356f187" containerID="f4b0bd5fafa269ded97db10939483dcdb1e9965d165abfaaab42b019235be00d" exitCode=0 Jun 02 01:54:38.445181 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:38.444830 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-59dcdf78f5-7qx6c" event={"ID":"a516b703-7511-4797-bcbe-763ca356f187","Type":"ContainerDied","Data":"f4b0bd5fafa269ded97db10939483dcdb1e9965d165abfaaab42b019235be00d"} Jun 02 01:54:39.449488 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:39.449453 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-59dcdf78f5-7qx6c" event={"ID":"a516b703-7511-4797-bcbe-763ca356f187","Type":"ContainerStarted","Data":"c9c46680f78f9b60a90266babffebee00074d92b63782de70f353921f4374655"} Jun 02 01:54:39.468783 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:39.468701 2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-59dcdf78f5-7qx6c" podStartSLOduration=3.298580793 podStartE2EDuration="3.468684623s" podCreationTimestamp="2026-06-02 01:54:36 +0000 UTC" firstStartedPulling="2026-06-02 01:54:38.445346769 +0000 UTC m=+796.091153389" lastFinishedPulling="2026-06-02 01:54:38.615450599 +0000 UTC m=+796.261257219" observedRunningTime="2026-06-02 01:54:39.467389415 +0000 UTC m=+797.113196057" watchObservedRunningTime="2026-06-02 01:54:39.468684623 +0000 UTC m=+797.114491264" Jun 02 01:54:46.607646 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:46.607609 2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-59dcdf78f5-7qx6c" Jun 02 01:54:46.607646 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:46.607653 2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-59dcdf78f5-7qx6c" Jun 02 01:54:46.620279 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:46.620254 2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-59dcdf78f5-7qx6c" Jun 02 01:54:47.485906 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:47.485874 2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-59dcdf78f5-7qx6c" Jun 02 01:54:48.366305 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:48.366270 2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-d8bdbcb5c-678sb"] Jun 02 01:54:48.370034 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:48.370016 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-d8bdbcb5c-678sb" Jun 02 01:54:48.372564 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:48.372543 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jun 02 01:54:48.378664 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:48.378641 2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-d8bdbcb5c-678sb"] Jun 02 01:54:48.461571 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:48.461534 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47e79c10-9df6-45a4-bf43-cbd637fcd810-kserve-provision-location\") pod \"premium-simulated-simulated-premium-kserve-d8bdbcb5c-678sb\" (UID: \"47e79c10-9df6-45a4-bf43-cbd637fcd810\") " pod="llm/premium-simulated-simulated-premium-kserve-d8bdbcb5c-678sb" Jun 02 01:54:48.461762 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:48.461595 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47e79c10-9df6-45a4-bf43-cbd637fcd810-model-cache\") pod \"premium-simulated-simulated-premium-kserve-d8bdbcb5c-678sb\" (UID: \"47e79c10-9df6-45a4-bf43-cbd637fcd810\") " pod="llm/premium-simulated-simulated-premium-kserve-d8bdbcb5c-678sb" Jun 02 01:54:48.461762 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:48.461620 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47e79c10-9df6-45a4-bf43-cbd637fcd810-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-d8bdbcb5c-678sb\" (UID: \"47e79c10-9df6-45a4-bf43-cbd637fcd810\") " pod="llm/premium-simulated-simulated-premium-kserve-d8bdbcb5c-678sb" Jun 02 01:54:48.461762 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:48.461653 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47e79c10-9df6-45a4-bf43-cbd637fcd810-home\") pod \"premium-simulated-simulated-premium-kserve-d8bdbcb5c-678sb\" (UID: \"47e79c10-9df6-45a4-bf43-cbd637fcd810\") " pod="llm/premium-simulated-simulated-premium-kserve-d8bdbcb5c-678sb" Jun 02 01:54:48.461762 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:48.461677 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thdqj\" (UniqueName: \"kubernetes.io/projected/47e79c10-9df6-45a4-bf43-cbd637fcd810-kube-api-access-thdqj\") pod \"premium-simulated-simulated-premium-kserve-d8bdbcb5c-678sb\" (UID: \"47e79c10-9df6-45a4-bf43-cbd637fcd810\") " pod="llm/premium-simulated-simulated-premium-kserve-d8bdbcb5c-678sb" Jun 02 01:54:48.461762 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:48.461705 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47e79c10-9df6-45a4-bf43-cbd637fcd810-dshm\") pod \"premium-simulated-simulated-premium-kserve-d8bdbcb5c-678sb\" (UID: \"47e79c10-9df6-45a4-bf43-cbd637fcd810\") " pod="llm/premium-simulated-simulated-premium-kserve-d8bdbcb5c-678sb" Jun 02 01:54:48.562824 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:48.562790 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47e79c10-9df6-45a4-bf43-cbd637fcd810-model-cache\") pod \"premium-simulated-simulated-premium-kserve-d8bdbcb5c-678sb\" (UID: \"47e79c10-9df6-45a4-bf43-cbd637fcd810\") " pod="llm/premium-simulated-simulated-premium-kserve-d8bdbcb5c-678sb" Jun 02 01:54:48.562824 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:48.562828 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47e79c10-9df6-45a4-bf43-cbd637fcd810-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-d8bdbcb5c-678sb\" (UID: \"47e79c10-9df6-45a4-bf43-cbd637fcd810\") " pod="llm/premium-simulated-simulated-premium-kserve-d8bdbcb5c-678sb" Jun 02 01:54:48.563063 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:48.562913 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47e79c10-9df6-45a4-bf43-cbd637fcd810-home\") pod \"premium-simulated-simulated-premium-kserve-d8bdbcb5c-678sb\" (UID: \"47e79c10-9df6-45a4-bf43-cbd637fcd810\") " pod="llm/premium-simulated-simulated-premium-kserve-d8bdbcb5c-678sb" Jun 02 01:54:48.563063 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:48.562943 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-thdqj\" (UniqueName: \"kubernetes.io/projected/47e79c10-9df6-45a4-bf43-cbd637fcd810-kube-api-access-thdqj\") pod \"premium-simulated-simulated-premium-kserve-d8bdbcb5c-678sb\" (UID: \"47e79c10-9df6-45a4-bf43-cbd637fcd810\") " pod="llm/premium-simulated-simulated-premium-kserve-d8bdbcb5c-678sb" Jun 02 01:54:48.563063 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:48.562968 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47e79c10-9df6-45a4-bf43-cbd637fcd810-dshm\") pod \"premium-simulated-simulated-premium-kserve-d8bdbcb5c-678sb\" (UID: \"47e79c10-9df6-45a4-bf43-cbd637fcd810\") " pod="llm/premium-simulated-simulated-premium-kserve-d8bdbcb5c-678sb" Jun 02 01:54:48.563261 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:48.563066 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47e79c10-9df6-45a4-bf43-cbd637fcd810-kserve-provision-location\") pod \"premium-simulated-simulated-premium-kserve-d8bdbcb5c-678sb\" (UID: \"47e79c10-9df6-45a4-bf43-cbd637fcd810\") " pod="llm/premium-simulated-simulated-premium-kserve-d8bdbcb5c-678sb" Jun 02 01:54:48.563318 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:48.563283 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47e79c10-9df6-45a4-bf43-cbd637fcd810-model-cache\") pod \"premium-simulated-simulated-premium-kserve-d8bdbcb5c-678sb\" (UID: \"47e79c10-9df6-45a4-bf43-cbd637fcd810\") " pod="llm/premium-simulated-simulated-premium-kserve-d8bdbcb5c-678sb" Jun 02 01:54:48.563378 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:48.563361 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47e79c10-9df6-45a4-bf43-cbd637fcd810-home\") pod \"premium-simulated-simulated-premium-kserve-d8bdbcb5c-678sb\" (UID: \"47e79c10-9df6-45a4-bf43-cbd637fcd810\") " pod="llm/premium-simulated-simulated-premium-kserve-d8bdbcb5c-678sb" Jun 02 01:54:48.563464 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:48.563442 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47e79c10-9df6-45a4-bf43-cbd637fcd810-kserve-provision-location\") pod \"premium-simulated-simulated-premium-kserve-d8bdbcb5c-678sb\" (UID: \"47e79c10-9df6-45a4-bf43-cbd637fcd810\") " pod="llm/premium-simulated-simulated-premium-kserve-d8bdbcb5c-678sb" Jun 02 01:54:48.565263 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:48.565239 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47e79c10-9df6-45a4-bf43-cbd637fcd810-dshm\") pod \"premium-simulated-simulated-premium-kserve-d8bdbcb5c-678sb\" (UID: \"47e79c10-9df6-45a4-bf43-cbd637fcd810\") " pod="llm/premium-simulated-simulated-premium-kserve-d8bdbcb5c-678sb" Jun 02 01:54:48.565433 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:48.565416 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47e79c10-9df6-45a4-bf43-cbd637fcd810-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-d8bdbcb5c-678sb\" (UID: \"47e79c10-9df6-45a4-bf43-cbd637fcd810\") " pod="llm/premium-simulated-simulated-premium-kserve-d8bdbcb5c-678sb" Jun 02 01:54:48.571845 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:48.571819 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thdqj\" (UniqueName: \"kubernetes.io/projected/47e79c10-9df6-45a4-bf43-cbd637fcd810-kube-api-access-thdqj\") pod \"premium-simulated-simulated-premium-kserve-d8bdbcb5c-678sb\" (UID: \"47e79c10-9df6-45a4-bf43-cbd637fcd810\") " pod="llm/premium-simulated-simulated-premium-kserve-d8bdbcb5c-678sb" Jun 02 01:54:48.681452 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:48.681414 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-d8bdbcb5c-678sb" Jun 02 01:54:48.810429 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:48.810402 2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-d8bdbcb5c-678sb"] Jun 02 01:54:48.812981 ip-10-0-135-117 kubenswrapper[2571]: W0602 01:54:48.812950 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47e79c10_9df6_45a4_bf43_cbd637fcd810.slice/crio-db56974fc468ca5fd28598052b07d754523fe7f0149e477ed6466f7c3005fa9c WatchSource:0}: Error finding container db56974fc468ca5fd28598052b07d754523fe7f0149e477ed6466f7c3005fa9c: Status 404 returned error can't find the container with id db56974fc468ca5fd28598052b07d754523fe7f0149e477ed6466f7c3005fa9c Jun 02 01:54:49.482714 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:49.482674 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-d8bdbcb5c-678sb" event={"ID":"47e79c10-9df6-45a4-bf43-cbd637fcd810","Type":"ContainerStarted","Data":"773b66b2034bf40631694c3774d88a0c67004f107742a8a110ec52b78d9ab91d"} Jun 02 01:54:49.482714 ip-10-0-135-117 kubenswrapper[2571]: I0602 01:54:49.482717 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-d8bdbcb5c-678sb" event={"ID":"47e79c10-9df6-45a4-bf43-cbd637fcd810","Type":"ContainerStarted","Data":"db56974fc468ca5fd28598052b07d754523fe7f0149e477ed6466f7c3005fa9c"} Jun 02 02:00:31.053860 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:00:31.053809 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-d8bdbcb5c-678sb_47e79c10-9df6-45a4-bf43-cbd637fcd810/storage-initializer/0.log" Jun 02 02:00:32.020877 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:00:32.020843 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-544d5c477f-xhxr8_08e0b510-bb35-47e4-beac-dac9dc53adf7/authorino/0.log" Jun 02 02:00:36.844829 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:00:36.844788 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-6b989f9fdb-nfrzb_3948f648-01e8-47e1-a1fc-e531b48f601b/maas-api/0.log" Jun 02 02:00:37.055892 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:00:37.055862 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7f759c8d56-bdxj7_30029d1b-9ffd-42d2-963a-a7ce6eee50b0/manager/0.log" Jun 02 02:00:37.274072 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:00:37.274042 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-76588b95-jpnzx_46600c1c-d993-4e0c-9304-e4f361bed002/manager/0.log" Jun 02 02:00:37.517840 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:00:37.517809 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6fdd9d74b4-frhbs_ae682085-80a3-41b9-8708-95f16511d75f/copy-manifests/0.log" Jun 02 02:00:37.531708 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:00:37.531621 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6fdd9d74b4-frhbs_ae682085-80a3-41b9-8708-95f16511d75f/manager/0.log" Jun 02 02:00:38.993988 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:00:38.993961 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-544d5c477f-xhxr8_08e0b510-bb35-47e4-beac-dac9dc53adf7/authorino/0.log" Jun 02 02:00:39.343221 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:00:39.343118 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-6cb54b5c86-wnnv2_34e43a88-2140-4348-901e-ffe294da6535/kuadrant-console-plugin/0.log" Jun 02 02:00:40.472373 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:00:40.472334 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5f795fc994-hjljs_396a0364-b4ad-49eb-8c61-0ed962f7d4d7/kube-auth-proxy/0.log" Jun 02 02:00:41.408817 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:00:41.408784 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-5489f4f4bc-wh9lm_88e4f184-ec91-493d-9fc3-b09fee7a72ce/storage-initializer/0.log" Jun 02 02:00:41.421046 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:00:41.421021 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-5489f4f4bc-wh9lm_88e4f184-ec91-493d-9fc3-b09fee7a72ce/main/0.log" Jun 02 02:00:41.762530 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:00:41.762432 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-59dcdf78f5-7qx6c_a516b703-7511-4797-bcbe-763ca356f187/storage-initializer/0.log" Jun 02 02:00:41.771733 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:00:41.771704 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-59dcdf78f5-7qx6c_a516b703-7511-4797-bcbe-763ca356f187/main/0.log" Jun 02 02:00:41.882116 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:00:41.882085 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-d8bdbcb5c-678sb_47e79c10-9df6-45a4-bf43-cbd637fcd810/storage-initializer/0.log" Jun 02 02:00:49.235366 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:00:49.235333 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-lqc45_7fd14a82-6157-4b43-ac1d-497b935341a5/global-pull-secret-syncer/0.log" Jun 02 02:00:49.375619 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:00:49.375587 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-bmd65_ab3aef63-c4c0-4b01-b26b-53ca0e39e7bb/konnectivity-agent/0.log" Jun 02 02:00:49.485903 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:00:49.485809 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-117.ec2.internal_dddc001fa74725ad60cc8708aa217f2d/haproxy/0.log" Jun 02 02:00:54.004676 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:00:54.004629 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-544d5c477f-xhxr8_08e0b510-bb35-47e4-beac-dac9dc53adf7/authorino/0.log" Jun 02 02:00:54.103125 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:00:54.103095 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-6cb54b5c86-wnnv2_34e43a88-2140-4348-901e-ffe294da6535/kuadrant-console-plugin/0.log" Jun 02 02:00:55.763365 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:00:55.763337 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7f4e3de2-1d34-464b-8dc3-dd008db7de61/alertmanager/0.log" Jun 02 02:00:55.803200 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:00:55.803174 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7f4e3de2-1d34-464b-8dc3-dd008db7de61/config-reloader/0.log" Jun 02 02:00:55.846253 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:00:55.846216 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7f4e3de2-1d34-464b-8dc3-dd008db7de61/kube-rbac-proxy-web/0.log" Jun 02 02:00:55.879027 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:00:55.879001 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7f4e3de2-1d34-464b-8dc3-dd008db7de61/kube-rbac-proxy/0.log" Jun 02 02:00:55.907766 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:00:55.907741 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7f4e3de2-1d34-464b-8dc3-dd008db7de61/kube-rbac-proxy-metric/0.log" Jun 02 02:00:55.936962 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:00:55.936934 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7f4e3de2-1d34-464b-8dc3-dd008db7de61/prom-label-proxy/0.log" Jun 02 02:00:55.972946 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:00:55.972918 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7f4e3de2-1d34-464b-8dc3-dd008db7de61/init-config-reloader/0.log" Jun 02 02:00:56.190662 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:00:56.190634 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5768fcc9dd-w9b6j_1b8379d9-608a-4c5b-a377-b4b69afb78b7/monitoring-plugin/0.log" Jun 02 02:00:56.338913 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:00:56.338885 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4rfv6_60142a30-6a11-4fca-9e87-30271db2c7f6/node-exporter/0.log" Jun 02 02:00:56.381352 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:00:56.381311 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4rfv6_60142a30-6a11-4fca-9e87-30271db2c7f6/kube-rbac-proxy/0.log" Jun 02 02:00:56.430289 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:00:56.430263 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4rfv6_60142a30-6a11-4fca-9e87-30271db2c7f6/init-textfile/0.log" Jun 02 02:00:57.019436 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:00:57.019404 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5b855cb479-9wn4b_d7e8c237-8e5c-44ac-8ade-bdab1fabd39a/telemeter-client/0.log" Jun 02 02:00:57.044572 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:00:57.044545 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5b855cb479-9wn4b_d7e8c237-8e5c-44ac-8ade-bdab1fabd39a/reload/0.log" Jun 02 02:00:57.074179 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:00:57.074144 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5b855cb479-9wn4b_d7e8c237-8e5c-44ac-8ade-bdab1fabd39a/kube-rbac-proxy/0.log" Jun 02 02:00:58.726702 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:00:58.726666 2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7ftlk/perf-node-gather-daemonset-rmp7f"] Jun 02 02:00:58.730098 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:00:58.730075 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7ftlk/perf-node-gather-daemonset-rmp7f" Jun 02 02:00:58.733032 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:00:58.733003 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7ftlk\"/\"openshift-service-ca.crt\"" Jun 02 02:00:58.733153 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:00:58.733110 2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7ftlk\"/\"kube-root-ca.crt\"" Jun 02 02:00:58.734098 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:00:58.734085 2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-7ftlk\"/\"default-dockercfg-gns69\"" Jun 02 02:00:58.775482 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:00:58.775446 2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7ftlk/perf-node-gather-daemonset-rmp7f"] Jun 02 02:00:58.799249 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:00:58.799218 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d9938cd2-513e-4e17-a155-e2094f1ff020-sys\") pod \"perf-node-gather-daemonset-rmp7f\" (UID: \"d9938cd2-513e-4e17-a155-e2094f1ff020\") " pod="openshift-must-gather-7ftlk/perf-node-gather-daemonset-rmp7f" Jun 02 02:00:58.799417 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:00:58.799270 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d9938cd2-513e-4e17-a155-e2094f1ff020-proc\") pod \"perf-node-gather-daemonset-rmp7f\" (UID: \"d9938cd2-513e-4e17-a155-e2094f1ff020\") " pod="openshift-must-gather-7ftlk/perf-node-gather-daemonset-rmp7f" Jun 02 02:00:58.799417 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:00:58.799297 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpvv8\" (UniqueName: \"kubernetes.io/projected/d9938cd2-513e-4e17-a155-e2094f1ff020-kube-api-access-wpvv8\") pod \"perf-node-gather-daemonset-rmp7f\" (UID: \"d9938cd2-513e-4e17-a155-e2094f1ff020\") " pod="openshift-must-gather-7ftlk/perf-node-gather-daemonset-rmp7f" Jun 02 02:00:58.799417 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:00:58.799364 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d9938cd2-513e-4e17-a155-e2094f1ff020-lib-modules\") pod \"perf-node-gather-daemonset-rmp7f\" (UID: \"d9938cd2-513e-4e17-a155-e2094f1ff020\") " pod="openshift-must-gather-7ftlk/perf-node-gather-daemonset-rmp7f" Jun 02 02:00:58.799417 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:00:58.799409 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d9938cd2-513e-4e17-a155-e2094f1ff020-podres\") pod \"perf-node-gather-daemonset-rmp7f\" (UID: \"d9938cd2-513e-4e17-a155-e2094f1ff020\") " pod="openshift-must-gather-7ftlk/perf-node-gather-daemonset-rmp7f" Jun 02 02:00:58.900533 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:00:58.900488 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d9938cd2-513e-4e17-a155-e2094f1ff020-sys\") pod \"perf-node-gather-daemonset-rmp7f\" (UID: \"d9938cd2-513e-4e17-a155-e2094f1ff020\") " pod="openshift-must-gather-7ftlk/perf-node-gather-daemonset-rmp7f" Jun 02 02:00:58.900707 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:00:58.900549 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d9938cd2-513e-4e17-a155-e2094f1ff020-proc\") pod \"perf-node-gather-daemonset-rmp7f\" (UID: \"d9938cd2-513e-4e17-a155-e2094f1ff020\") " pod="openshift-must-gather-7ftlk/perf-node-gather-daemonset-rmp7f" Jun 02 02:00:58.900707 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:00:58.900571 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d9938cd2-513e-4e17-a155-e2094f1ff020-sys\") pod \"perf-node-gather-daemonset-rmp7f\" (UID: \"d9938cd2-513e-4e17-a155-e2094f1ff020\") " pod="openshift-must-gather-7ftlk/perf-node-gather-daemonset-rmp7f" Jun 02 02:00:58.900707 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:00:58.900572 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wpvv8\" (UniqueName: \"kubernetes.io/projected/d9938cd2-513e-4e17-a155-e2094f1ff020-kube-api-access-wpvv8\") pod \"perf-node-gather-daemonset-rmp7f\" (UID: \"d9938cd2-513e-4e17-a155-e2094f1ff020\") " pod="openshift-must-gather-7ftlk/perf-node-gather-daemonset-rmp7f" Jun 02 02:00:58.900707 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:00:58.900636 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d9938cd2-513e-4e17-a155-e2094f1ff020-proc\") pod \"perf-node-gather-daemonset-rmp7f\" (UID: \"d9938cd2-513e-4e17-a155-e2094f1ff020\") " pod="openshift-must-gather-7ftlk/perf-node-gather-daemonset-rmp7f" Jun 02 02:00:58.900707 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:00:58.900699 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d9938cd2-513e-4e17-a155-e2094f1ff020-lib-modules\") pod \"perf-node-gather-daemonset-rmp7f\" (UID: \"d9938cd2-513e-4e17-a155-e2094f1ff020\") " pod="openshift-must-gather-7ftlk/perf-node-gather-daemonset-rmp7f" Jun 02 02:00:58.900877 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:00:58.900750 2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d9938cd2-513e-4e17-a155-e2094f1ff020-podres\") pod \"perf-node-gather-daemonset-rmp7f\" (UID: \"d9938cd2-513e-4e17-a155-e2094f1ff020\") " pod="openshift-must-gather-7ftlk/perf-node-gather-daemonset-rmp7f" Jun 02 02:00:58.900877 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:00:58.900829 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d9938cd2-513e-4e17-a155-e2094f1ff020-lib-modules\") pod \"perf-node-gather-daemonset-rmp7f\" (UID: \"d9938cd2-513e-4e17-a155-e2094f1ff020\") " pod="openshift-must-gather-7ftlk/perf-node-gather-daemonset-rmp7f" Jun 02 02:00:58.900877 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:00:58.900870 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d9938cd2-513e-4e17-a155-e2094f1ff020-podres\") pod \"perf-node-gather-daemonset-rmp7f\" (UID: \"d9938cd2-513e-4e17-a155-e2094f1ff020\") " pod="openshift-must-gather-7ftlk/perf-node-gather-daemonset-rmp7f" Jun 02 02:00:58.910386 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:00:58.910360 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpvv8\" (UniqueName: \"kubernetes.io/projected/d9938cd2-513e-4e17-a155-e2094f1ff020-kube-api-access-wpvv8\") pod \"perf-node-gather-daemonset-rmp7f\" (UID: \"d9938cd2-513e-4e17-a155-e2094f1ff020\") " pod="openshift-must-gather-7ftlk/perf-node-gather-daemonset-rmp7f" Jun 02 02:00:59.039990 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:00:59.039898 2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7ftlk/perf-node-gather-daemonset-rmp7f" Jun 02 02:00:59.175815 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:00:59.175786 2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7ftlk/perf-node-gather-daemonset-rmp7f"] Jun 02 02:00:59.177422 ip-10-0-135-117 kubenswrapper[2571]: W0602 02:00:59.177397 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd9938cd2_513e_4e17_a155_e2094f1ff020.slice/crio-5f81fb90bae7046c1f4b85aebde8aa1d4c83d1ffa723f8103cfe8c7c239c4ecf WatchSource:0}: Error finding container 5f81fb90bae7046c1f4b85aebde8aa1d4c83d1ffa723f8103cfe8c7c239c4ecf: Status 404 returned error can't find the container with id 5f81fb90bae7046c1f4b85aebde8aa1d4c83d1ffa723f8103cfe8c7c239c4ecf Jun 02 02:00:59.179143 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:00:59.179104 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jun 02 02:00:59.718376 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:00:59.718341 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7ftlk/perf-node-gather-daemonset-rmp7f" event={"ID":"d9938cd2-513e-4e17-a155-e2094f1ff020","Type":"ContainerStarted","Data":"c107b03702d3ff9edfcb4bdd59bf2e207a7f4e80d1327b39322a760f4519e8dd"} Jun 02 02:00:59.718376 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:00:59.718380 2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7ftlk/perf-node-gather-daemonset-rmp7f" event={"ID":"d9938cd2-513e-4e17-a155-e2094f1ff020","Type":"ContainerStarted","Data":"5f81fb90bae7046c1f4b85aebde8aa1d4c83d1ffa723f8103cfe8c7c239c4ecf"} Jun 02 02:00:59.718606 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:00:59.718417 2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-7ftlk/perf-node-gather-daemonset-rmp7f" Jun 02 02:00:59.738629 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:00:59.738555 2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-7ftlk/perf-node-gather-daemonset-rmp7f" podStartSLOduration=1.738541723 podStartE2EDuration="1.738541723s" podCreationTimestamp="2026-06-02 02:00:58 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 02:00:59.73637951 +0000 UTC m=+1177.382186152" watchObservedRunningTime="2026-06-02 02:00:59.738541723 +0000 UTC m=+1177.384348365" Jun 02 02:01:00.847846 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:01:00.847813 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fs4kq_97195e1b-54fc-432f-a909-88dac67a6235/dns/0.log" Jun 02 02:01:00.875012 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:01:00.874983 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fs4kq_97195e1b-54fc-432f-a909-88dac67a6235/kube-rbac-proxy/0.log" Jun 02 02:01:00.984715 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:01:00.984685 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nm89v_5c45e264-330c-4438-bdc1-817a0a8b1b77/dns-node-resolver/0.log" Jun 02 02:01:01.537029 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:01:01.536996 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-657f6fdbb8-cntl6_ebbbf316-4688-4404-b302-8cc241ec36b7/registry/0.log" Jun 02 02:01:01.590971 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:01:01.590938 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6rzvj_1fc82c63-2678-486d-9393-c3ea7f27abce/node-ca/0.log" Jun 02 02:01:02.663213 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:01:02.663179 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5f795fc994-hjljs_396a0364-b4ad-49eb-8c61-0ed962f7d4d7/kube-auth-proxy/0.log" Jun 02 02:01:03.419840 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:01:03.419813 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-m4mvd_e5cbb2d6-6afa-4be3-9aab-2f3bd30125bd/serve-healthcheck-canary/0.log" Jun 02 02:01:04.125825 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:01:04.125794 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vmqlc_62166777-842d-49dc-9215-7b408dff79c9/kube-rbac-proxy/0.log" Jun 02 02:01:04.149050 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:01:04.149027 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vmqlc_62166777-842d-49dc-9215-7b408dff79c9/exporter/0.log" Jun 02 02:01:04.174683 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:01:04.174654 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vmqlc_62166777-842d-49dc-9215-7b408dff79c9/extractor/0.log" Jun 02 02:01:05.733546 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:01:05.733515 2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-7ftlk/perf-node-gather-daemonset-rmp7f" Jun 02 02:01:06.386218 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:01:06.386185 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-6b989f9fdb-nfrzb_3948f648-01e8-47e1-a1fc-e531b48f601b/maas-api/0.log" Jun 02 02:01:06.437106 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:01:06.437074 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7f759c8d56-bdxj7_30029d1b-9ffd-42d2-963a-a7ce6eee50b0/manager/0.log" Jun 02 02:01:06.510029 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:01:06.509997 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-76588b95-jpnzx_46600c1c-d993-4e0c-9304-e4f361bed002/manager/0.log" Jun 02 02:01:06.642331 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:01:06.642304 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6fdd9d74b4-frhbs_ae682085-80a3-41b9-8708-95f16511d75f/manager/0.log" Jun 02 02:01:06.662652 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:01:06.662626 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6fdd9d74b4-frhbs_ae682085-80a3-41b9-8708-95f16511d75f/copy-manifests/0.log" Jun 02 02:01:08.461702 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:01:08.461675 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-74db8bbd6d-9dx5c_85e911a4-7429-4fce-9e76-b339a2a78c38/manager/0.log" Jun 02 02:01:14.955808 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:01:14.955773 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-7drhb_ba65c8da-80cd-4fda-9746-dd851a3a06be/kube-multus/0.log" Jun 02 02:01:15.228207 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:01:15.228094 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tglfq_ae3e1499-5744-4270-b94d-f292d30bd1b2/kube-multus-additional-cni-plugins/0.log" Jun 02 02:01:15.264681 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:01:15.264646 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tglfq_ae3e1499-5744-4270-b94d-f292d30bd1b2/egress-router-binary-copy/0.log" Jun 02 02:01:15.293920 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:01:15.293891 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tglfq_ae3e1499-5744-4270-b94d-f292d30bd1b2/cni-plugins/0.log" Jun 02 02:01:15.321974 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:01:15.321947 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tglfq_ae3e1499-5744-4270-b94d-f292d30bd1b2/bond-cni-plugin/0.log" Jun 02 02:01:15.348193 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:01:15.348165 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tglfq_ae3e1499-5744-4270-b94d-f292d30bd1b2/routeoverride-cni/0.log" Jun 02 02:01:15.380441 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:01:15.380415 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tglfq_ae3e1499-5744-4270-b94d-f292d30bd1b2/whereabouts-cni-bincopy/0.log" Jun 02 02:01:15.408572 ip-10-0-135-117 kubenswrapper[2571]: I0602 02:01:15.408545 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tglfq_ae3e1499-5744-4270-b94d-f292d30bd1b2/whereabouts-cni/0.log"