Apr 16 13:57:10.047026 ip-10-0-136-114 systemd[1]: Starting Kubernetes Kubelet... Apr 16 13:57:10.521304 ip-10-0-136-114 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Apr 16 13:57:10.521304 ip-10-0-136-114 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Apr 16 13:57:10.521304 ip-10-0-136-114 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Apr 16 13:57:10.521304 ip-10-0-136-114 kubenswrapper[2574]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Apr 16 13:57:10.521304 ip-10-0-136-114 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Apr 16 13:57:10.524785 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.524683 2574 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Apr 16 13:57:10.528816 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.528794 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Apr 16 13:57:10.528816 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.528812 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Apr 16 13:57:10.528816 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.528816 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Apr 16 13:57:10.528816 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.528819 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Apr 16 13:57:10.528816 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.528823 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS Apr 16 13:57:10.529002 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.528826 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Apr 16 13:57:10.529002 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.528829 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Apr 16 13:57:10.529002 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.528831 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Apr 16 13:57:10.529002 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.528834 2574 feature_gate.go:328] unrecognized feature gate: Example Apr 16 13:57:10.529002 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.528837 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Apr 16 13:57:10.529002 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.528839 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Apr 16 13:57:10.529002 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.528842 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Apr 16 13:57:10.529002 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.528844 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Apr 16 13:57:10.529002 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.528847 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Apr 16 13:57:10.529002 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.528849 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Apr 16 13:57:10.529002 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.528852 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Apr 16 13:57:10.529002 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.528855 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Apr 16 13:57:10.529002 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.528857 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Apr 16 13:57:10.529002 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.528860 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Apr 16 13:57:10.529002 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.528863 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Apr 16 13:57:10.529002 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.528866 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Apr 16 13:57:10.529002 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.528869 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Apr 16 13:57:10.529002 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.528872 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Apr 16 13:57:10.529002 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.528874 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Apr 16 13:57:10.529002 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.528877 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Apr 16 13:57:10.529501 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.528879 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Apr 16 13:57:10.529501 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.528882 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Apr 16 13:57:10.529501 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.528892 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Apr 16 13:57:10.529501 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.528895 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Apr 16 13:57:10.529501 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.528897 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Apr 16 13:57:10.529501 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.528899 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Apr 16 13:57:10.529501 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.528903 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Apr 16 13:57:10.529501 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.528905 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Apr 16 13:57:10.529501 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.528908 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Apr 16 13:57:10.529501 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.528910 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Apr 16 13:57:10.529501 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.528913 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Apr 16 13:57:10.529501 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.528916 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Apr 16 13:57:10.529501 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.528919 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Apr 16 13:57:10.529501 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.528923 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI Apr 16 13:57:10.529501 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.528926 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Apr 16 13:57:10.529501 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.528930 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Apr 16 13:57:10.529501 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.528935 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Apr 16 13:57:10.529501 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.528938 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Apr 16 13:57:10.529501 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.528941 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Apr 16 13:57:10.529993 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.528945 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Apr 16 13:57:10.529993 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.528948 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Apr 16 13:57:10.529993 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.528952 2574 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController Apr 16 13:57:10.529993 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.528955 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Apr 16 13:57:10.529993 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.528959 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Apr 16 13:57:10.529993 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.528963 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Apr 16 13:57:10.529993 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.528966 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Apr 16 13:57:10.529993 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.528969 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Apr 16 13:57:10.529993 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.528972 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations Apr 16 13:57:10.529993 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.528975 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Apr 16 13:57:10.529993 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.528978 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Apr 16 13:57:10.529993 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.528981 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Apr 16 13:57:10.529993 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.528983 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Apr 16 13:57:10.529993 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.528986 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Apr 16 13:57:10.529993 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.528989 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Apr 16 13:57:10.529993 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.528991 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Apr 16 13:57:10.529993 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.528994 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Apr 16 13:57:10.529993 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.528997 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Apr 16 13:57:10.529993 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529007 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Apr 16 13:57:10.530444 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529009 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Apr 16 13:57:10.530444 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529012 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Apr 16 13:57:10.530444 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529043 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Apr 16 13:57:10.530444 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529047 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Apr 16 13:57:10.530444 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529050 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Apr 16 13:57:10.530444 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529053 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Apr 16 13:57:10.530444 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529057 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Apr 16 13:57:10.530444 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529061 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Apr 16 13:57:10.530444 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529064 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Apr 16 13:57:10.530444 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529068 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Apr 16 13:57:10.530444 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529071 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Apr 16 13:57:10.530444 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529073 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Apr 16 13:57:10.530444 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529076 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Apr 16 13:57:10.530444 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529079 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Apr 16 13:57:10.530444 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529082 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Apr 16 13:57:10.530444 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529086 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Apr 16 13:57:10.530444 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529089 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Apr 16 13:57:10.530444 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529091 2574 feature_gate.go:328] unrecognized feature gate: Example2 Apr 16 13:57:10.530444 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529094 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Apr 16 13:57:10.530444 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529096 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Apr 16 13:57:10.530982 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529099 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Apr 16 13:57:10.530982 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529102 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Apr 16 13:57:10.530982 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529104 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Apr 16 13:57:10.530982 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529493 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Apr 16 13:57:10.530982 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529499 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Apr 16 13:57:10.530982 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529502 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Apr 16 13:57:10.530982 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529505 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Apr 16 13:57:10.530982 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529508 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Apr 16 13:57:10.530982 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529511 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Apr 16 13:57:10.530982 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529514 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Apr 16 13:57:10.530982 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529517 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Apr 16 13:57:10.530982 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529520 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Apr 16 13:57:10.530982 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529523 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Apr 16 13:57:10.530982 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529525 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Apr 16 13:57:10.530982 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529528 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Apr 16 13:57:10.530982 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529531 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Apr 16 13:57:10.530982 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529534 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Apr 16 13:57:10.530982 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529537 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Apr 16 13:57:10.530982 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529540 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Apr 16 13:57:10.531451 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529543 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Apr 16 13:57:10.531451 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529545 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Apr 16 13:57:10.531451 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529548 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Apr 16 13:57:10.531451 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529551 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Apr 16 13:57:10.531451 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529553 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Apr 16 13:57:10.531451 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529556 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Apr 16 13:57:10.531451 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529559 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Apr 16 13:57:10.531451 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529561 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Apr 16 13:57:10.531451 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529565 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Apr 16 13:57:10.531451 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529567 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Apr 16 13:57:10.531451 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529572 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Apr 16 13:57:10.531451 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529575 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Apr 16 13:57:10.531451 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529578 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Apr 16 13:57:10.531451 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529582 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Apr 16 13:57:10.531451 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529585 2574 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController Apr 16 13:57:10.531451 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529587 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Apr 16 13:57:10.531451 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529590 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Apr 16 13:57:10.531451 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529593 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Apr 16 13:57:10.531451 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529596 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Apr 16 13:57:10.531971 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529599 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Apr 16 13:57:10.531971 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529601 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Apr 16 13:57:10.531971 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529604 2574 feature_gate.go:328] unrecognized feature gate: Example Apr 16 13:57:10.531971 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529607 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Apr 16 13:57:10.531971 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529610 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Apr 16 13:57:10.531971 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529613 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS Apr 16 13:57:10.531971 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529616 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Apr 16 13:57:10.531971 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529618 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Apr 16 13:57:10.531971 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529621 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Apr 16 13:57:10.531971 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529624 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Apr 16 13:57:10.531971 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529626 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Apr 16 13:57:10.531971 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529639 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Apr 16 13:57:10.531971 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529643 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Apr 16 13:57:10.531971 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529647 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Apr 16 13:57:10.531971 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529650 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Apr 16 13:57:10.531971 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529653 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Apr 16 13:57:10.531971 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529657 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Apr 16 13:57:10.531971 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529659 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Apr 16 13:57:10.531971 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529662 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Apr 16 13:57:10.531971 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529665 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Apr 16 13:57:10.532454 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529667 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Apr 16 13:57:10.532454 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529670 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Apr 16 13:57:10.532454 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529673 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Apr 16 13:57:10.532454 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529675 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Apr 16 13:57:10.532454 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529678 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Apr 16 13:57:10.532454 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529681 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Apr 16 13:57:10.532454 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529683 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Apr 16 13:57:10.532454 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529686 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Apr 16 13:57:10.532454 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529688 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Apr 16 13:57:10.532454 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529691 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Apr 16 13:57:10.532454 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529693 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Apr 16 13:57:10.532454 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529696 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Apr 16 13:57:10.532454 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529698 2574 feature_gate.go:328] unrecognized feature gate: Example2 Apr 16 13:57:10.532454 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529703 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Apr 16 13:57:10.532454 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529706 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Apr 16 13:57:10.532454 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529708 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Apr 16 13:57:10.532454 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529711 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Apr 16 13:57:10.532454 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529713 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Apr 16 13:57:10.532454 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529716 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI Apr 16 13:57:10.532454 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529718 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Apr 16 13:57:10.532961 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529721 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Apr 16 13:57:10.532961 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529723 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations Apr 16 13:57:10.532961 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529726 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Apr 16 13:57:10.532961 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529728 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Apr 16 13:57:10.532961 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529731 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Apr 16 13:57:10.532961 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529734 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Apr 16 13:57:10.532961 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529736 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Apr 16 13:57:10.532961 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529753 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Apr 16 13:57:10.532961 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529755 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Apr 16 13:57:10.532961 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529758 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Apr 16 13:57:10.532961 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.529761 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Apr 16 13:57:10.532961 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531268 2574 flags.go:64] FLAG: --address="0.0.0.0" Apr 16 13:57:10.532961 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531277 2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Apr 16 13:57:10.532961 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531285 2574 flags.go:64] FLAG: --anonymous-auth="true" Apr 16 13:57:10.532961 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531290 2574 flags.go:64] FLAG: --application-metrics-count-limit="100" Apr 16 13:57:10.532961 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531295 2574 flags.go:64] FLAG: --authentication-token-webhook="false" Apr 16 13:57:10.532961 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531299 2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Apr 16 13:57:10.532961 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531303 2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Apr 16 13:57:10.532961 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531308 2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Apr 16 13:57:10.532961 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531311 2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Apr 16 13:57:10.532961 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531314 2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Apr 16 13:57:10.533468 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531318 2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Apr 16 13:57:10.533468 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531321 2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Apr 16 13:57:10.533468 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531324 2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Apr 16 13:57:10.533468 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531327 2574 flags.go:64] FLAG: --cgroup-root="" Apr 16 13:57:10.533468 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531331 2574 flags.go:64] FLAG: --cgroups-per-qos="true" Apr 16 13:57:10.533468 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531334 2574 flags.go:64] FLAG: --client-ca-file="" Apr 16 13:57:10.533468 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531336 2574 flags.go:64] FLAG: --cloud-config="" Apr 16 13:57:10.533468 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531339 2574 flags.go:64] FLAG: --cloud-provider="external" Apr 16 13:57:10.533468 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531342 2574 flags.go:64] FLAG: --cluster-dns="[]" Apr 16 13:57:10.533468 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531346 2574 flags.go:64] FLAG: --cluster-domain="" Apr 16 13:57:10.533468 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531349 2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Apr 16 13:57:10.533468 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531352 2574 flags.go:64] FLAG: --config-dir="" Apr 16 13:57:10.533468 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531355 2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Apr 16 13:57:10.533468 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531359 2574 flags.go:64] FLAG: --container-log-max-files="5" Apr 16 13:57:10.533468 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531362 2574 flags.go:64] FLAG: --container-log-max-size="10Mi" Apr 16 13:57:10.533468 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531366 2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Apr 16 13:57:10.533468 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531369 2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Apr 16 13:57:10.533468 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531373 2574 flags.go:64] FLAG: --containerd-namespace="k8s.io" Apr 16 13:57:10.533468 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531376 2574 flags.go:64] FLAG: --contention-profiling="false" Apr 16 13:57:10.533468 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531379 2574 flags.go:64] FLAG: --cpu-cfs-quota="true" Apr 16 13:57:10.533468 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531382 2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Apr 16 13:57:10.533468 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531385 2574 flags.go:64] FLAG: --cpu-manager-policy="none" Apr 16 13:57:10.533468 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531388 2574 flags.go:64] FLAG: --cpu-manager-policy-options="" Apr 16 13:57:10.533468 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531392 2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Apr 16 13:57:10.533468 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531395 2574 flags.go:64] FLAG: --enable-controller-attach-detach="true" Apr 16 13:57:10.534109 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531398 2574 flags.go:64] FLAG: --enable-debugging-handlers="true" Apr 16 13:57:10.534109 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531401 2574 flags.go:64] FLAG: --enable-load-reader="false" Apr 16 13:57:10.534109 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531405 2574 flags.go:64] FLAG: --enable-server="true" Apr 16 13:57:10.534109 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531408 2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Apr 16 13:57:10.534109 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531412 2574 flags.go:64] FLAG: --event-burst="100" Apr 16 13:57:10.534109 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531415 2574 flags.go:64] FLAG: --event-qps="50" Apr 16 13:57:10.534109 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531419 2574 flags.go:64] FLAG: --event-storage-age-limit="default=0" Apr 16 13:57:10.534109 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531422 2574 flags.go:64] FLAG: --event-storage-event-limit="default=0" Apr 16 13:57:10.534109 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531425 2574 flags.go:64] FLAG: --eviction-hard="" Apr 16 13:57:10.534109 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531429 2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Apr 16 13:57:10.534109 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531432 2574 flags.go:64] FLAG: --eviction-minimum-reclaim="" Apr 16 13:57:10.534109 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531435 2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Apr 16 13:57:10.534109 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531439 2574 flags.go:64] FLAG: --eviction-soft="" Apr 16 13:57:10.534109 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531442 2574 flags.go:64] FLAG: --eviction-soft-grace-period="" Apr 16 13:57:10.534109 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531445 2574 flags.go:64] FLAG: --exit-on-lock-contention="false" Apr 16 13:57:10.534109 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531447 2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Apr 16 13:57:10.534109 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531450 2574 flags.go:64] FLAG: --experimental-mounter-path="" Apr 16 13:57:10.534109 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531454 2574 flags.go:64] FLAG: --fail-cgroupv1="false" Apr 16 13:57:10.534109 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531457 2574 flags.go:64] FLAG: --fail-swap-on="true" Apr 16 13:57:10.534109 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531460 2574 flags.go:64] FLAG: --feature-gates="" Apr 16 13:57:10.534109 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531464 2574 flags.go:64] FLAG: --file-check-frequency="20s" Apr 16 13:57:10.534109 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531467 2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Apr 16 13:57:10.534109 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531470 2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Apr 16 13:57:10.534109 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531476 2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Apr 16 13:57:10.534109 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531479 2574 flags.go:64] FLAG: --healthz-port="10248" Apr 16 13:57:10.534109 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531482 2574 flags.go:64] FLAG: --help="false" Apr 16 13:57:10.534751 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531485 2574 flags.go:64] FLAG: --hostname-override="ip-10-0-136-114.ec2.internal" Apr 16 13:57:10.534751 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531488 2574 flags.go:64] FLAG: --housekeeping-interval="10s" Apr 16 13:57:10.534751 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531491 2574 flags.go:64] FLAG: --http-check-frequency="20s" Apr 16 13:57:10.534751 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531495 2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Apr 16 13:57:10.534751 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531498 2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Apr 16 13:57:10.534751 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531502 2574 flags.go:64] FLAG: --image-gc-high-threshold="85" Apr 16 13:57:10.534751 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531504 2574 flags.go:64] FLAG: --image-gc-low-threshold="80" Apr 16 13:57:10.534751 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531507 2574 flags.go:64] FLAG: --image-service-endpoint="" Apr 16 13:57:10.534751 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531510 2574 flags.go:64] FLAG: --kernel-memcg-notification="false" Apr 16 13:57:10.534751 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531514 2574 flags.go:64] FLAG: --kube-api-burst="100" Apr 16 13:57:10.534751 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531517 2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Apr 16 13:57:10.534751 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531520 2574 flags.go:64] FLAG: --kube-api-qps="50" Apr 16 13:57:10.534751 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531523 2574 flags.go:64] FLAG: --kube-reserved="" Apr 16 13:57:10.534751 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531526 2574 flags.go:64] FLAG: --kube-reserved-cgroup="" Apr 16 13:57:10.534751 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531529 2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Apr 16 13:57:10.534751 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531532 2574 flags.go:64] FLAG: --kubelet-cgroups="" Apr 16 13:57:10.534751 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531535 2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Apr 16 13:57:10.534751 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531538 2574 flags.go:64] FLAG: --lock-file="" Apr 16 13:57:10.534751 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531541 2574 flags.go:64] FLAG: --log-cadvisor-usage="false" Apr 16 13:57:10.534751 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531544 2574 flags.go:64] FLAG: --log-flush-frequency="5s" Apr 16 13:57:10.534751 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531547 2574 flags.go:64] FLAG: --log-json-info-buffer-size="0" Apr 16 13:57:10.534751 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531552 2574 flags.go:64] FLAG: --log-json-split-stream="false" Apr 16 13:57:10.534751 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531555 2574 flags.go:64] FLAG: --log-text-info-buffer-size="0" Apr 16 13:57:10.535314 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531558 2574 flags.go:64] FLAG: --log-text-split-stream="false" Apr 16 13:57:10.535314 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531561 2574 flags.go:64] FLAG: --logging-format="text" Apr 16 13:57:10.535314 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531564 2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Apr 16 13:57:10.535314 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531567 2574 flags.go:64] FLAG: --make-iptables-util-chains="true" Apr 16 13:57:10.535314 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531570 2574 flags.go:64] FLAG: --manifest-url="" Apr 16 13:57:10.535314 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531572 2574 flags.go:64] FLAG: --manifest-url-header="" Apr 16 13:57:10.535314 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531578 2574 flags.go:64] FLAG: --max-housekeeping-interval="15s" Apr 16 13:57:10.535314 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531581 2574 flags.go:64] FLAG: --max-open-files="1000000" Apr 16 13:57:10.535314 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531585 2574 flags.go:64] FLAG: --max-pods="110" Apr 16 13:57:10.535314 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531588 2574 flags.go:64] FLAG: --maximum-dead-containers="-1" Apr 16 13:57:10.535314 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531592 2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Apr 16 13:57:10.535314 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531595 2574 flags.go:64] FLAG: --memory-manager-policy="None" Apr 16 13:57:10.535314 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531598 2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Apr 16 13:57:10.535314 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531601 2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Apr 16 13:57:10.535314 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531603 2574 flags.go:64] FLAG: --node-ip="0.0.0.0" Apr 16 13:57:10.535314 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531607 2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Apr 16 13:57:10.535314 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531614 2574 flags.go:64] FLAG: --node-status-max-images="50" Apr 16 13:57:10.535314 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531617 2574 flags.go:64] FLAG: --node-status-update-frequency="10s" Apr 16 13:57:10.535314 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531620 2574 flags.go:64] FLAG: --oom-score-adj="-999" Apr 16 13:57:10.535314 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531624 2574 flags.go:64] FLAG: --pod-cidr="" Apr 16 13:57:10.535314 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531627 2574 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dc76bab72f320de3d4105c90d73c4fb139c09e20ce0fa8dcbc0cb59920d27dec" Apr 16 13:57:10.535314 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531632 2574 flags.go:64] FLAG: --pod-manifest-path="" Apr 16 13:57:10.535314 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531634 2574 flags.go:64] FLAG: --pod-max-pids="-1" Apr 16 13:57:10.535314 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531638 2574 flags.go:64] FLAG: --pods-per-core="0" Apr 16 13:57:10.535908 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531641 2574 flags.go:64] FLAG: --port="10250" Apr 16 13:57:10.535908 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531644 2574 flags.go:64] FLAG: --protect-kernel-defaults="false" Apr 16 13:57:10.535908 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531647 2574 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-07c79e0d77f098719" Apr 16 13:57:10.535908 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531650 2574 flags.go:64] FLAG: --qos-reserved="" Apr 16 13:57:10.535908 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531653 2574 flags.go:64] FLAG: --read-only-port="10255" Apr 16 13:57:10.535908 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531656 2574 flags.go:64] FLAG: --register-node="true" Apr 16 13:57:10.535908 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531659 2574 flags.go:64] FLAG: --register-schedulable="true" Apr 16 13:57:10.535908 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531662 2574 flags.go:64] FLAG: --register-with-taints="" Apr 16 13:57:10.535908 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531665 2574 flags.go:64] FLAG: --registry-burst="10" Apr 16 13:57:10.535908 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531668 2574 flags.go:64] FLAG: --registry-qps="5" Apr 16 13:57:10.535908 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531671 2574 flags.go:64] FLAG: --reserved-cpus="" Apr 16 13:57:10.535908 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531673 2574 flags.go:64] FLAG: --reserved-memory="" Apr 16 13:57:10.535908 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531677 2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Apr 16 13:57:10.535908 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531680 2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Apr 16 13:57:10.535908 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531684 2574 flags.go:64] FLAG: --rotate-certificates="false" Apr 16 13:57:10.535908 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531687 2574 flags.go:64] FLAG: --rotate-server-certificates="false" Apr 16 13:57:10.535908 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531690 2574 flags.go:64] FLAG: --runonce="false" Apr 16 13:57:10.535908 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531693 2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Apr 16 13:57:10.535908 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531696 2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Apr 16 13:57:10.535908 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531699 2574 flags.go:64] FLAG: --seccomp-default="false" Apr 16 13:57:10.535908 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531702 2574 flags.go:64] FLAG: --serialize-image-pulls="true" Apr 16 13:57:10.535908 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531705 2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Apr 16 13:57:10.535908 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531708 2574 flags.go:64] FLAG: --storage-driver-db="cadvisor" Apr 16 13:57:10.535908 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531711 2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Apr 16 13:57:10.535908 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531714 2574 flags.go:64] FLAG: --storage-driver-password="root" Apr 16 13:57:10.535908 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531717 2574 flags.go:64] FLAG: --storage-driver-secure="false" Apr 16 13:57:10.536524 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531720 2574 flags.go:64] FLAG: --storage-driver-table="stats" Apr 16 13:57:10.536524 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531723 2574 flags.go:64] FLAG: --storage-driver-user="root" Apr 16 13:57:10.536524 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531726 2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Apr 16 13:57:10.536524 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531729 2574 flags.go:64] FLAG: --sync-frequency="1m0s" Apr 16 13:57:10.536524 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531732 2574 flags.go:64] FLAG: --system-cgroups="" Apr 16 13:57:10.536524 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531735 2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Apr 16 13:57:10.536524 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531758 2574 flags.go:64] FLAG: --system-reserved-cgroup="" Apr 16 13:57:10.536524 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531763 2574 flags.go:64] FLAG: --tls-cert-file="" Apr 16 13:57:10.536524 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531768 2574 flags.go:64] FLAG: --tls-cipher-suites="[]" Apr 16 13:57:10.536524 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531773 2574 flags.go:64] FLAG: --tls-min-version="" Apr 16 13:57:10.536524 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531776 2574 flags.go:64] FLAG: --tls-private-key-file="" Apr 16 13:57:10.536524 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531778 2574 flags.go:64] FLAG: --topology-manager-policy="none" Apr 16 13:57:10.536524 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531781 2574 flags.go:64] FLAG: --topology-manager-policy-options="" Apr 16 13:57:10.536524 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531784 2574 flags.go:64] FLAG: --topology-manager-scope="container" Apr 16 13:57:10.536524 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531788 2574 flags.go:64] FLAG: --v="2" Apr 16 13:57:10.536524 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531792 2574 flags.go:64] FLAG: --version="false" Apr 16 13:57:10.536524 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531796 2574 flags.go:64] FLAG: --vmodule="" Apr 16 13:57:10.536524 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531800 2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Apr 16 13:57:10.536524 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.531804 2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Apr 16 13:57:10.536524 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.531904 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Apr 16 13:57:10.536524 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.531910 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Apr 16 13:57:10.536524 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.531913 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Apr 16 13:57:10.536524 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.531916 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Apr 16 13:57:10.536524 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.531919 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Apr 16 13:57:10.537145 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.531922 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Apr 16 13:57:10.537145 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.531925 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Apr 16 13:57:10.537145 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.531928 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Apr 16 13:57:10.537145 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.531931 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Apr 16 13:57:10.537145 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.531934 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Apr 16 13:57:10.537145 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.531938 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Apr 16 13:57:10.537145 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.531941 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Apr 16 13:57:10.537145 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.531943 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Apr 16 13:57:10.537145 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.531946 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Apr 16 13:57:10.537145 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.531949 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Apr 16 13:57:10.537145 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.531952 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Apr 16 13:57:10.537145 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.531955 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Apr 16 13:57:10.537145 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.531959 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Apr 16 13:57:10.537145 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.531962 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Apr 16 13:57:10.537145 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.531965 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Apr 16 13:57:10.537145 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.531968 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Apr 16 13:57:10.537145 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.531973 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Apr 16 13:57:10.537145 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.531975 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Apr 16 13:57:10.537145 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.531978 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Apr 16 13:57:10.537627 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.531981 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Apr 16 13:57:10.537627 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.531983 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Apr 16 13:57:10.537627 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.531986 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Apr 16 13:57:10.537627 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.531989 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI Apr 16 13:57:10.537627 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.531992 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Apr 16 13:57:10.537627 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.531994 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Apr 16 13:57:10.537627 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.531997 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Apr 16 13:57:10.537627 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.532000 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Apr 16 13:57:10.537627 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.532002 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Apr 16 13:57:10.537627 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.532006 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations Apr 16 13:57:10.537627 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.532008 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Apr 16 13:57:10.537627 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.532011 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS Apr 16 13:57:10.537627 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.532014 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Apr 16 13:57:10.537627 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.532016 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Apr 16 13:57:10.537627 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.532019 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Apr 16 13:57:10.537627 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.532022 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Apr 16 13:57:10.537627 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.532024 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Apr 16 13:57:10.537627 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.532027 2574 feature_gate.go:328] unrecognized feature gate: Example Apr 16 13:57:10.537627 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.532029 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Apr 16 13:57:10.537627 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.532032 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Apr 16 13:57:10.538220 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.532034 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Apr 16 13:57:10.538220 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.532037 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Apr 16 13:57:10.538220 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.532040 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Apr 16 13:57:10.538220 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.532043 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Apr 16 13:57:10.538220 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.532045 2574 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController Apr 16 13:57:10.538220 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.532048 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Apr 16 13:57:10.538220 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.532051 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Apr 16 13:57:10.538220 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.532053 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Apr 16 13:57:10.538220 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.532056 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Apr 16 13:57:10.538220 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.532060 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Apr 16 13:57:10.538220 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.532063 2574 feature_gate.go:328] unrecognized feature gate: Example2 Apr 16 13:57:10.538220 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.532065 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Apr 16 13:57:10.538220 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.532068 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Apr 16 13:57:10.538220 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.532070 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Apr 16 13:57:10.538220 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.532073 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Apr 16 13:57:10.538220 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.532075 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Apr 16 13:57:10.538220 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.532078 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Apr 16 13:57:10.538220 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.532081 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Apr 16 13:57:10.538220 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.532083 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Apr 16 13:57:10.538220 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.532086 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Apr 16 13:57:10.538727 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.532088 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Apr 16 13:57:10.538727 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.532091 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Apr 16 13:57:10.538727 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.532094 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Apr 16 13:57:10.538727 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.532097 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Apr 16 13:57:10.538727 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.532099 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Apr 16 13:57:10.538727 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.532102 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Apr 16 13:57:10.538727 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.532105 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Apr 16 13:57:10.538727 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.532107 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Apr 16 13:57:10.538727 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.532110 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Apr 16 13:57:10.538727 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.532113 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Apr 16 13:57:10.538727 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.532115 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Apr 16 13:57:10.538727 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.532117 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Apr 16 13:57:10.538727 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.532122 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Apr 16 13:57:10.538727 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.532126 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Apr 16 13:57:10.538727 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.532128 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Apr 16 13:57:10.538727 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.532131 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Apr 16 13:57:10.538727 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.532133 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Apr 16 13:57:10.538727 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.532136 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Apr 16 13:57:10.538727 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.532139 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Apr 16 13:57:10.538727 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.532141 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Apr 16 13:57:10.539238 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.532144 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Apr 16 13:57:10.539238 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.532148 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Apr 16 13:57:10.539238 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.533031 2574 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]} Apr 16 13:57:10.540366 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.540345 2574 server.go:530] "Kubelet version" kubeletVersion="v1.33.9" Apr 16 13:57:10.540404 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.540367 2574 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Apr 16 13:57:10.540434 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540415 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Apr 16 13:57:10.540434 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540420 2574 feature_gate.go:328] unrecognized feature gate: Example2 Apr 16 13:57:10.540434 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540423 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Apr 16 13:57:10.540434 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540427 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Apr 16 13:57:10.540434 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540430 2574 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController Apr 16 13:57:10.540434 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540433 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Apr 16 13:57:10.540434 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540436 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI Apr 16 13:57:10.540618 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540439 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Apr 16 13:57:10.540618 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540441 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Apr 16 13:57:10.540618 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540445 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Apr 16 13:57:10.540618 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540447 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Apr 16 13:57:10.540618 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540450 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Apr 16 13:57:10.540618 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540453 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Apr 16 13:57:10.540618 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540456 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Apr 16 13:57:10.540618 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540458 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Apr 16 13:57:10.540618 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540461 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Apr 16 13:57:10.540618 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540464 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Apr 16 13:57:10.540618 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540467 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Apr 16 13:57:10.540618 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540469 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Apr 16 13:57:10.540618 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540478 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Apr 16 13:57:10.540618 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540480 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Apr 16 13:57:10.540618 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540483 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Apr 16 13:57:10.540618 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540486 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Apr 16 13:57:10.540618 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540488 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Apr 16 13:57:10.540618 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540491 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Apr 16 13:57:10.540618 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540494 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Apr 16 13:57:10.540618 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540496 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Apr 16 13:57:10.541128 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540499 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Apr 16 13:57:10.541128 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540502 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Apr 16 13:57:10.541128 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540504 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Apr 16 13:57:10.541128 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540507 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Apr 16 13:57:10.541128 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540512 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Apr 16 13:57:10.541128 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540514 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Apr 16 13:57:10.541128 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540517 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Apr 16 13:57:10.541128 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540520 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Apr 16 13:57:10.541128 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540523 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Apr 16 13:57:10.541128 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540525 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Apr 16 13:57:10.541128 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540528 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Apr 16 13:57:10.541128 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540530 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations Apr 16 13:57:10.541128 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540533 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Apr 16 13:57:10.541128 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540536 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Apr 16 13:57:10.541128 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540541 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Apr 16 13:57:10.541128 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540545 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Apr 16 13:57:10.541128 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540548 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Apr 16 13:57:10.541128 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540551 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Apr 16 13:57:10.541128 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540554 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Apr 16 13:57:10.541128 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540556 2574 feature_gate.go:328] unrecognized feature gate: Example Apr 16 13:57:10.541614 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540559 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Apr 16 13:57:10.541614 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540562 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Apr 16 13:57:10.541614 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540565 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Apr 16 13:57:10.541614 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540567 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Apr 16 13:57:10.541614 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540570 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Apr 16 13:57:10.541614 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540573 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Apr 16 13:57:10.541614 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540576 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Apr 16 13:57:10.541614 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540580 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Apr 16 13:57:10.541614 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540583 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Apr 16 13:57:10.541614 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540587 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Apr 16 13:57:10.541614 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540590 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Apr 16 13:57:10.541614 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540593 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Apr 16 13:57:10.541614 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540595 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Apr 16 13:57:10.541614 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540598 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Apr 16 13:57:10.541614 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540601 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Apr 16 13:57:10.541614 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540603 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Apr 16 13:57:10.541614 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540606 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Apr 16 13:57:10.541614 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540609 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS Apr 16 13:57:10.541614 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540612 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Apr 16 13:57:10.541614 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540615 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Apr 16 13:57:10.542296 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540617 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Apr 16 13:57:10.542296 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540620 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Apr 16 13:57:10.542296 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540622 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Apr 16 13:57:10.542296 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540625 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Apr 16 13:57:10.542296 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540628 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Apr 16 13:57:10.542296 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540633 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Apr 16 13:57:10.542296 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540635 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Apr 16 13:57:10.542296 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540638 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Apr 16 13:57:10.542296 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540640 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Apr 16 13:57:10.542296 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540643 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Apr 16 13:57:10.542296 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540645 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Apr 16 13:57:10.542296 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540648 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Apr 16 13:57:10.542296 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540651 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Apr 16 13:57:10.542296 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540653 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Apr 16 13:57:10.542296 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540656 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Apr 16 13:57:10.542296 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540658 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Apr 16 13:57:10.542296 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540661 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Apr 16 13:57:10.542296 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540663 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Apr 16 13:57:10.542296 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540666 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Apr 16 13:57:10.542769 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.540671 2574 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]} Apr 16 13:57:10.542769 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540780 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Apr 16 13:57:10.542769 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540785 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Apr 16 13:57:10.542769 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540788 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Apr 16 13:57:10.542769 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540791 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Apr 16 13:57:10.542769 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540793 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Apr 16 13:57:10.542769 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540796 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Apr 16 13:57:10.542769 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540799 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Apr 16 13:57:10.542769 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540801 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Apr 16 13:57:10.542769 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540804 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Apr 16 13:57:10.542769 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540807 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Apr 16 13:57:10.542769 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540810 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Apr 16 13:57:10.542769 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540813 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Apr 16 13:57:10.542769 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540815 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Apr 16 13:57:10.542769 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540818 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Apr 16 13:57:10.543136 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540821 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Apr 16 13:57:10.543136 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540825 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Apr 16 13:57:10.543136 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540828 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Apr 16 13:57:10.543136 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540831 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Apr 16 13:57:10.543136 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540834 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Apr 16 13:57:10.543136 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540836 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Apr 16 13:57:10.543136 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540839 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Apr 16 13:57:10.543136 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540841 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Apr 16 13:57:10.543136 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540844 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Apr 16 13:57:10.543136 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540847 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Apr 16 13:57:10.543136 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540849 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Apr 16 13:57:10.543136 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540852 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Apr 16 13:57:10.543136 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540854 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Apr 16 13:57:10.543136 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540857 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Apr 16 13:57:10.543136 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540859 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Apr 16 13:57:10.543136 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540862 2574 feature_gate.go:328] unrecognized feature gate: Example2 Apr 16 13:57:10.543136 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540865 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Apr 16 13:57:10.543136 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540867 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Apr 16 13:57:10.543136 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540870 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Apr 16 13:57:10.543136 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540872 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Apr 16 13:57:10.543608 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540875 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Apr 16 13:57:10.543608 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540877 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Apr 16 13:57:10.543608 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540880 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Apr 16 13:57:10.543608 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540882 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Apr 16 13:57:10.543608 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540885 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations Apr 16 13:57:10.543608 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540887 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Apr 16 13:57:10.543608 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540890 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Apr 16 13:57:10.543608 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540893 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Apr 16 13:57:10.543608 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540896 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Apr 16 13:57:10.543608 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540899 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Apr 16 13:57:10.543608 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540901 2574 feature_gate.go:328] unrecognized feature gate: Example Apr 16 13:57:10.543608 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540904 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Apr 16 13:57:10.543608 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540907 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Apr 16 13:57:10.543608 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540909 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Apr 16 13:57:10.543608 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540912 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Apr 16 13:57:10.543608 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540914 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Apr 16 13:57:10.543608 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540917 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Apr 16 13:57:10.543608 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540919 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Apr 16 13:57:10.543608 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540922 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Apr 16 13:57:10.543608 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540924 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Apr 16 13:57:10.544116 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540927 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Apr 16 13:57:10.544116 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540929 2574 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController Apr 16 13:57:10.544116 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540932 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Apr 16 13:57:10.544116 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540934 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Apr 16 13:57:10.544116 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540937 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Apr 16 13:57:10.544116 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540939 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Apr 16 13:57:10.544116 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540942 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Apr 16 13:57:10.544116 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540945 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Apr 16 13:57:10.544116 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540947 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Apr 16 13:57:10.544116 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540950 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Apr 16 13:57:10.544116 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540952 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Apr 16 13:57:10.544116 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540956 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Apr 16 13:57:10.544116 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540959 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Apr 16 13:57:10.544116 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540962 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Apr 16 13:57:10.544116 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540965 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Apr 16 13:57:10.544116 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540968 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Apr 16 13:57:10.544116 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540971 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Apr 16 13:57:10.544116 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540973 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Apr 16 13:57:10.544116 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540976 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Apr 16 13:57:10.544568 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540979 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS Apr 16 13:57:10.544568 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540981 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI Apr 16 13:57:10.544568 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540984 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Apr 16 13:57:10.544568 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540986 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Apr 16 13:57:10.544568 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540989 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Apr 16 13:57:10.544568 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540991 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Apr 16 13:57:10.544568 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540994 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Apr 16 13:57:10.544568 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540997 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Apr 16 13:57:10.544568 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.540999 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Apr 16 13:57:10.544568 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.541002 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Apr 16 13:57:10.544568 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.541004 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Apr 16 13:57:10.544568 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.541007 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Apr 16 13:57:10.544568 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:10.541010 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Apr 16 13:57:10.544568 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.541014 2574 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]} Apr 16 13:57:10.544568 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.541858 2574 server.go:962] "Client rotation is on, will bootstrap in background" Apr 16 13:57:10.544979 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.544965 2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Apr 16 13:57:10.546707 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.546694 2574 server.go:1019] "Starting client certificate rotation" Apr 16 13:57:10.546818 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.546801 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Apr 16 13:57:10.547811 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.547797 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Apr 16 13:57:10.576015 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.575995 2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Apr 16 13:57:10.582102 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.582079 2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Apr 16 13:57:10.594407 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.594391 2574 log.go:25] "Validated CRI v1 runtime API" Apr 16 13:57:10.599241 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.599226 2574 log.go:25] "Validated CRI v1 image API" Apr 16 13:57:10.600370 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.600353 2574 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Apr 16 13:57:10.604642 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.604624 2574 fs.go:135] Filesystem UUIDs: map[4af7f48a-2325-4dc7-99d7-26b54312825d:/dev/nvme0n1p3 6b416544-2d47-4b62-9870-4d177a15d8b8:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Apr 16 13:57:10.604715 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.604641 2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Apr 16 13:57:10.608697 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.608676 2574 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Apr 16 13:57:10.610457 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.610354 2574 manager.go:217] Machine: {Timestamp:2026-04-16 13:57:10.608342945 +0000 UTC m=+0.438077807 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098064 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2dc6be10c9bd6fd411d9a6b1990831 SystemUUID:ec2dc6be-10c9-bd6f-d411-d9a6b1990831 BootID:f5970e0f-df4c-47cd-b5fd-2ce89cb30b63 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6098944 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:8a:62:b3:d6:2b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:8a:62:b3:d6:2b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:de:9a:72:49:4b:d2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Apr 16 13:57:10.611039 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.611028 2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Apr 16 13:57:10.611126 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.611115 2574 manager.go:233] Version: {KernelVersion:5.14.0-570.104.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260401-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Apr 16 13:57:10.612147 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.612123 2574 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Apr 16 13:57:10.612278 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.612149 2574 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-114.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Apr 16 13:57:10.612319 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.612287 2574 topology_manager.go:138] "Creating topology manager with none policy" Apr 16 13:57:10.612319 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.612294 2574 container_manager_linux.go:306] "Creating device plugin manager" Apr 16 13:57:10.612319 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.612311 2574 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Apr 16 13:57:10.612400 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.612324 2574 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Apr 16 13:57:10.614414 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.614402 2574 state_mem.go:36] "Initialized new in-memory state store" Apr 16 13:57:10.614519 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.614510 2574 server.go:1267] "Using root directory" path="/var/lib/kubelet" Apr 16 13:57:10.617368 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.617357 2574 kubelet.go:491] "Attempting to sync node with API server" Apr 16 13:57:10.617419 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.617372 2574 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests" Apr 16 13:57:10.617419 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.617389 2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Apr 16 13:57:10.617419 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.617398 2574 kubelet.go:397] "Adding apiserver pod source" Apr 16 13:57:10.617419 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.617407 2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Apr 16 13:57:10.618395 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.618383 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Apr 16 13:57:10.618441 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.618401 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Apr 16 13:57:10.621642 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.621628 2574 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.10-2.rhaos4.20.gita4d0894.el9" apiVersion="v1" Apr 16 13:57:10.635607 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.635588 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-56zdw" Apr 16 13:57:10.635607 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.635606 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-56zdw" Apr 16 13:57:10.635927 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.635913 2574 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Apr 16 13:57:10.637758 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.637724 2574 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Apr 16 13:57:10.637835 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.637771 2574 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Apr 16 13:57:10.637835 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.637782 2574 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Apr 16 13:57:10.637835 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.637792 2574 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Apr 16 13:57:10.637835 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.637808 2574 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Apr 16 13:57:10.637835 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.637817 2574 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret" Apr 16 13:57:10.637835 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.637827 2574 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Apr 16 13:57:10.637835 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.637835 2574 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Apr 16 13:57:10.638117 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.637847 2574 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc" Apr 16 13:57:10.638117 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.637857 2574 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Apr 16 13:57:10.638117 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.637871 2574 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected" Apr 16 13:57:10.638117 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.637884 2574 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Apr 16 13:57:10.638819 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.638803 2574 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi" Apr 16 13:57:10.638884 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.638838 2574 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image" Apr 16 13:57:10.642926 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.642911 2574 watchdog_linux.go:99] "Systemd watchdog is not enabled" Apr 16 13:57:10.642999 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.642956 2574 server.go:1295] "Started kubelet" Apr 16 13:57:10.643086 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.643047 2574 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Apr 16 13:57:10.643177 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.643137 2574 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Apr 16 13:57:10.643213 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.643197 2574 server_v1.go:47] "podresources" method="list" useActivePods=true Apr 16 13:57:10.643865 ip-10-0-136-114 systemd[1]: Started Kubernetes Kubelet. Apr 16 13:57:10.644405 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.644210 2574 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Apr 16 13:57:10.644992 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.644973 2574 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Apr 16 13:57:10.646213 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.646196 2574 server.go:317] "Adding debug handlers to kubelet server" Apr 16 13:57:10.647897 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.647879 2574 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Apr 16 13:57:10.648894 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.648868 2574 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-136-114.ec2.internal" not found Apr 16 13:57:10.649923 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.649908 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Apr 16 13:57:10.650507 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.650491 2574 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Apr 16 13:57:10.651360 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.651342 2574 factory.go:55] Registering systemd factory Apr 16 13:57:10.651441 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.651364 2574 factory.go:223] Registration of the systemd container factory successfully Apr 16 13:57:10.651638 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.651621 2574 desired_state_of_world_populator.go:150] "Desired state populator starts to run" Apr 16 13:57:10.651712 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.651646 2574 volume_manager.go:295] "The desired_state_of_world populator starts" Apr 16 13:57:10.651712 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.651663 2574 volume_manager.go:297] "Starting Kubelet Volume Manager" Apr 16 13:57:10.651712 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.651697 2574 factory.go:153] Registering CRI-O factory Apr 16 13:57:10.651712 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.651709 2574 factory.go:223] Registration of the crio container factory successfully Apr 16 13:57:10.651914 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.651789 2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Apr 16 13:57:10.651914 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.651817 2574 reconstruct.go:97] "Volume reconstruction finished" Apr 16 13:57:10.651914 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.651827 2574 reconciler.go:26] "Reconciler: start to sync state" Apr 16 13:57:10.651914 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.651831 2574 factory.go:103] Registering Raw factory Apr 16 13:57:10.651914 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.651848 2574 manager.go:1196] Started watching for new ooms in manager Apr 16 13:57:10.652116 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:10.651906 2574 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-136-114.ec2.internal\" not found" Apr 16 13:57:10.652287 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.652275 2574 manager.go:319] Starting recovery of all containers Apr 16 13:57:10.653050 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.653031 2574 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Apr 16 13:57:10.653324 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:10.653235 2574 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Apr 16 13:57:10.656963 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:10.656941 2574 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-136-114.ec2.internal\" not found" node="ip-10-0-136-114.ec2.internal" Apr 16 13:57:10.664519 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.664368 2574 manager.go:324] Recovery completed Apr 16 13:57:10.665109 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.665096 2574 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-136-114.ec2.internal" not found Apr 16 13:57:10.668589 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.668577 2574 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" Apr 16 13:57:10.670263 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.670247 2574 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-114.ec2.internal" event="NodeHasSufficientMemory" Apr 16 13:57:10.670319 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.670276 2574 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-114.ec2.internal" event="NodeHasNoDiskPressure" Apr 16 13:57:10.670319 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.670288 2574 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-114.ec2.internal" event="NodeHasSufficientPID" Apr 16 13:57:10.670729 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.670716 2574 cpu_manager.go:222] "Starting CPU manager" policy="none" Apr 16 13:57:10.670729 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.670726 2574 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Apr 16 13:57:10.670814 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.670755 2574 state_mem.go:36] "Initialized new in-memory state store" Apr 16 13:57:10.673152 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.673141 2574 policy_none.go:49] "None policy: Start" Apr 16 13:57:10.673192 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.673156 2574 memory_manager.go:186] "Starting memorymanager" policy="None" Apr 16 13:57:10.673192 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.673166 2574 state_mem.go:35] "Initializing new in-memory state store" Apr 16 13:57:10.702281 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.702267 2574 manager.go:341] "Starting Device Plugin manager" Apr 16 13:57:10.702360 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:10.702304 2574 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Apr 16 13:57:10.702360 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.702314 2574 server.go:85] "Starting device plugin registration server" Apr 16 13:57:10.702550 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.702537 2574 eviction_manager.go:189] "Eviction manager: starting control loop" Apr 16 13:57:10.702611 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.702553 2574 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Apr 16 13:57:10.702678 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.702657 2574 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Apr 16 13:57:10.702732 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.702727 2574 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Apr 16 13:57:10.702794 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.702735 2574 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Apr 16 13:57:10.703179 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:10.703156 2574 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Apr 16 13:57:10.703265 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:10.703204 2574 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-114.ec2.internal\" not found" Apr 16 13:57:10.721952 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.721930 2574 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-136-114.ec2.internal" not found Apr 16 13:57:10.802891 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.802826 2574 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" Apr 16 13:57:10.802891 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.802831 2574 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4" Apr 16 13:57:10.803982 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.803641 2574 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-114.ec2.internal" event="NodeHasSufficientMemory" Apr 16 13:57:10.803982 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.803670 2574 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-114.ec2.internal" event="NodeHasNoDiskPressure" Apr 16 13:57:10.803982 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.803683 2574 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-114.ec2.internal" event="NodeHasSufficientPID" Apr 16 13:57:10.803982 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.803710 2574 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-114.ec2.internal" Apr 16 13:57:10.804275 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.804260 2574 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6" Apr 16 13:57:10.804316 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.804287 2574 status_manager.go:230] "Starting to sync pod status with apiserver" Apr 16 13:57:10.804316 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.804307 2574 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Apr 16 13:57:10.804316 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.804314 2574 kubelet.go:2451] "Starting kubelet main sync loop" Apr 16 13:57:10.804413 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:10.804350 2574 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Apr 16 13:57:10.806880 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.806864 2574 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Apr 16 13:57:10.812405 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.812391 2574 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-114.ec2.internal" Apr 16 13:57:10.812461 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:10.812411 2574 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-114.ec2.internal\": node \"ip-10-0-136-114.ec2.internal\" not found" Apr 16 13:57:10.905971 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.905941 2574 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-114.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-114.ec2.internal"] Apr 16 13:57:10.908726 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.908711 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-114.ec2.internal" Apr 16 13:57:10.908799 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.908733 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-114.ec2.internal" Apr 16 13:57:10.935304 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.935287 2574 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-114.ec2.internal" Apr 16 13:57:10.939665 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.939652 2574 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-114.ec2.internal" Apr 16 13:57:10.952057 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.952034 2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Apr 16 13:57:10.953658 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:10.953643 2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Apr 16 13:57:11.054084 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.054026 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0db6df7e5746d57207c9078241175f3c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-114.ec2.internal\" (UID: \"0db6df7e5746d57207c9078241175f3c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-114.ec2.internal" Apr 16 13:57:11.054084 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.054052 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/031b74a3d41a1b92a3331318a0772b2d-config\") pod \"kube-apiserver-proxy-ip-10-0-136-114.ec2.internal\" (UID: \"031b74a3d41a1b92a3331318a0772b2d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-114.ec2.internal" Apr 16 13:57:11.054084 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.054068 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0db6df7e5746d57207c9078241175f3c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-114.ec2.internal\" (UID: \"0db6df7e5746d57207c9078241175f3c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-114.ec2.internal" Apr 16 13:57:11.154655 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.154627 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0db6df7e5746d57207c9078241175f3c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-114.ec2.internal\" (UID: \"0db6df7e5746d57207c9078241175f3c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-114.ec2.internal" Apr 16 13:57:11.154812 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.154597 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0db6df7e5746d57207c9078241175f3c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-114.ec2.internal\" (UID: \"0db6df7e5746d57207c9078241175f3c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-114.ec2.internal" Apr 16 13:57:11.154812 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.154715 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0db6df7e5746d57207c9078241175f3c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-114.ec2.internal\" (UID: \"0db6df7e5746d57207c9078241175f3c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-114.ec2.internal" Apr 16 13:57:11.154812 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.154734 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/031b74a3d41a1b92a3331318a0772b2d-config\") pod \"kube-apiserver-proxy-ip-10-0-136-114.ec2.internal\" (UID: \"031b74a3d41a1b92a3331318a0772b2d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-114.ec2.internal" Apr 16 13:57:11.154812 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.154784 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/031b74a3d41a1b92a3331318a0772b2d-config\") pod \"kube-apiserver-proxy-ip-10-0-136-114.ec2.internal\" (UID: \"031b74a3d41a1b92a3331318a0772b2d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-114.ec2.internal" Apr 16 13:57:11.154812 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.154799 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0db6df7e5746d57207c9078241175f3c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-114.ec2.internal\" (UID: \"0db6df7e5746d57207c9078241175f3c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-114.ec2.internal" Apr 16 13:57:11.255750 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.255726 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-114.ec2.internal" Apr 16 13:57:11.255852 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.255788 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-114.ec2.internal" Apr 16 13:57:11.545611 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.545533 2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Apr 16 13:57:11.546144 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.545698 2574 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Apr 16 13:57:11.546144 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.545720 2574 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Apr 16 13:57:11.546144 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.545736 2574 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Apr 16 13:57:11.618382 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.618352 2574 apiserver.go:52] "Watching apiserver" Apr 16 13:57:11.628162 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.628141 2574 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Apr 16 13:57:11.628485 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.628466 2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-99w5m","kube-system/konnectivity-agent-sflxw","kube-system/kube-apiserver-proxy-ip-10-0-136-114.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27wm7","openshift-cluster-node-tuning-operator/tuned-6srr9","openshift-multus/multus-additional-cni-plugins-rnvkv","openshift-multus/network-metrics-daemon-5fh28","openshift-network-diagnostics/network-check-target-h4r6r","openshift-network-operator/iptables-alerter-v2ntl","openshift-dns/node-resolver-x4qcz","openshift-image-registry/node-ca-brqdj","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-114.ec2.internal","openshift-multus/multus-8js6c"] Apr 16 13:57:11.631030 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.631005 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" Apr 16 13:57:11.632045 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.632028 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sflxw" Apr 16 13:57:11.632174 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.632160 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27wm7" Apr 16 13:57:11.633419 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.633407 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6srr9" Apr 16 13:57:11.633649 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.633634 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Apr 16 13:57:11.634077 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.634057 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Apr 16 13:57:11.634077 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.634068 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Apr 16 13:57:11.634193 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.634071 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Apr 16 13:57:11.634193 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.634166 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Apr 16 13:57:11.634640 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.634620 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rnvkv" Apr 16 13:57:11.634958 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.634916 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Apr 16 13:57:11.635082 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.635068 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Apr 16 13:57:11.635148 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.635089 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Apr 16 13:57:11.635203 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.635164 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Apr 16 13:57:11.635203 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.635197 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2jf46\"" Apr 16 13:57:11.635323 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.635306 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Apr 16 13:57:11.635437 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.635422 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-dbr77\"" Apr 16 13:57:11.635514 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.635499 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Apr 16 13:57:11.635568 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.635503 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-k9nz4\"" Apr 16 13:57:11.635836 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.635793 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5fh28" Apr 16 13:57:11.635919 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.635872 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Apr 16 13:57:11.635919 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:11.635880 2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5fh28" podUID="2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e" Apr 16 13:57:11.636013 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.635962 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rjz29\"" Apr 16 13:57:11.636013 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.636000 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Apr 16 13:57:11.636516 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.636503 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Apr 16 13:57:11.636566 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.636531 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-fjrgw\"" Apr 16 13:57:11.637001 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.636987 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4r6r" Apr 16 13:57:11.637047 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:11.637034 2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4r6r" podUID="df2f75cb-df70-4c82-b805-f93bcc7e8030" Apr 16 13:57:11.637088 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.637072 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-04-15 13:52:10 +0000 UTC" deadline="2028-01-19 09:17:19.344210942 +0000 UTC" Apr 16 13:57:11.637121 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.637089 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15427h20m7.707123838s" Apr 16 13:57:11.637307 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.637289 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Apr 16 13:57:11.637397 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.637297 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Apr 16 13:57:11.637397 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.637340 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Apr 16 13:57:11.637397 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.637366 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Apr 16 13:57:11.638425 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.638411 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-v2ntl" Apr 16 13:57:11.639543 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.639529 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-x4qcz" Apr 16 13:57:11.640222 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.640205 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5s4x2\"" Apr 16 13:57:11.640397 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.640384 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Apr 16 13:57:11.640459 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.640412 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Apr 16 13:57:11.640692 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.640678 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-brqdj" Apr 16 13:57:11.641040 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.641026 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Apr 16 13:57:11.641205 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.641190 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9c8gz\"" Apr 16 13:57:11.641456 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.641441 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Apr 16 13:57:11.641528 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.641475 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Apr 16 13:57:11.641706 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.641690 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8js6c" Apr 16 13:57:11.642435 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.642420 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Apr 16 13:57:11.642586 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.642573 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Apr 16 13:57:11.642683 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.642668 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Apr 16 13:57:11.642761 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.642707 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7vmbj\"" Apr 16 13:57:11.643358 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.643342 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Apr 16 13:57:11.643430 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.643380 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-fb74c\"" Apr 16 13:57:11.650160 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.650145 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Apr 16 13:57:11.652982 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.652967 2574 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world" Apr 16 13:57:11.658715 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.658696 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bbd94c79-236c-4a37-b6ad-384d38491b9e-var-lib-kubelet\") pod \"tuned-6srr9\" (UID: \"bbd94c79-236c-4a37-b6ad-384d38491b9e\") " pod="openshift-cluster-node-tuning-operator/tuned-6srr9" Apr 16 13:57:11.658822 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.658722 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/48194990-0de3-45a9-acc5-6fcc4c2760d0-serviceca\") pod \"node-ca-brqdj\" (UID: \"48194990-0de3-45a9-acc5-6fcc4c2760d0\") " pod="openshift-image-registry/node-ca-brqdj" Apr 16 13:57:11.658822 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.658752 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b83df07-d346-4caa-b358-294994a3d89b-multus-cni-dir\") pod \"multus-8js6c\" (UID: \"5b83df07-d346-4caa-b358-294994a3d89b\") " pod="openshift-multus/multus-8js6c" Apr 16 13:57:11.658822 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.658779 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c18aa469-e518-4e2b-9d80-85e7e1d8ed94-etc-openvswitch\") pod \"ovnkube-node-99w5m\" (UID: \"c18aa469-e518-4e2b-9d80-85e7e1d8ed94\") " pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" Apr 16 13:57:11.658981 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.658827 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c18aa469-e518-4e2b-9d80-85e7e1d8ed94-run-openvswitch\") pod \"ovnkube-node-99w5m\" (UID: \"c18aa469-e518-4e2b-9d80-85e7e1d8ed94\") " pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" Apr 16 13:57:11.658981 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.658843 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d29c2bfc-de60-442e-a4cb-615b6688e211-sys-fs\") pod \"aws-ebs-csi-driver-node-27wm7\" (UID: \"d29c2bfc-de60-442e-a4cb-615b6688e211\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27wm7" Apr 16 13:57:11.658981 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.658864 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e-metrics-certs\") pod \"network-metrics-daemon-5fh28\" (UID: \"2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e\") " pod="openshift-multus/network-metrics-daemon-5fh28" Apr 16 13:57:11.658981 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.658881 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjn79\" (UniqueName: \"kubernetes.io/projected/bbd94c79-236c-4a37-b6ad-384d38491b9e-kube-api-access-zjn79\") pod \"tuned-6srr9\" (UID: \"bbd94c79-236c-4a37-b6ad-384d38491b9e\") " pod="openshift-cluster-node-tuning-operator/tuned-6srr9" Apr 16 13:57:11.658981 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.658902 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b83df07-d346-4caa-b358-294994a3d89b-host-var-lib-cni-bin\") pod \"multus-8js6c\" (UID: \"5b83df07-d346-4caa-b358-294994a3d89b\") " pod="openshift-multus/multus-8js6c" Apr 16 13:57:11.658981 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.658925 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5b83df07-d346-4caa-b358-294994a3d89b-multus-daemon-config\") pod \"multus-8js6c\" (UID: \"5b83df07-d346-4caa-b358-294994a3d89b\") " pod="openshift-multus/multus-8js6c" Apr 16 13:57:11.658981 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.658948 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d29c2bfc-de60-442e-a4cb-615b6688e211-etc-selinux\") pod \"aws-ebs-csi-driver-node-27wm7\" (UID: \"d29c2bfc-de60-442e-a4cb-615b6688e211\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27wm7" Apr 16 13:57:11.658981 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.658970 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7d2abdf5-4927-45c4-9386-af7dc0ed8d90-iptables-alerter-script\") pod \"iptables-alerter-v2ntl\" (UID: \"7d2abdf5-4927-45c4-9386-af7dc0ed8d90\") " pod="openshift-network-operator/iptables-alerter-v2ntl" Apr 16 13:57:11.659203 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.658993 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/48194990-0de3-45a9-acc5-6fcc4c2760d0-host\") pod \"node-ca-brqdj\" (UID: \"48194990-0de3-45a9-acc5-6fcc4c2760d0\") " pod="openshift-image-registry/node-ca-brqdj" Apr 16 13:57:11.659203 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.659008 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5b83df07-d346-4caa-b358-294994a3d89b-host-run-multus-certs\") pod \"multus-8js6c\" (UID: \"5b83df07-d346-4caa-b358-294994a3d89b\") " pod="openshift-multus/multus-8js6c" Apr 16 13:57:11.659203 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.659028 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c18aa469-e518-4e2b-9d80-85e7e1d8ed94-var-lib-openvswitch\") pod \"ovnkube-node-99w5m\" (UID: \"c18aa469-e518-4e2b-9d80-85e7e1d8ed94\") " pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" Apr 16 13:57:11.659203 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.659044 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c18aa469-e518-4e2b-9d80-85e7e1d8ed94-host-cni-netd\") pod \"ovnkube-node-99w5m\" (UID: \"c18aa469-e518-4e2b-9d80-85e7e1d8ed94\") " pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" Apr 16 13:57:11.659203 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.659057 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz86b\" (UniqueName: \"kubernetes.io/projected/c590b6a7-f5a5-476f-a0bd-fb8e8853b8cb-kube-api-access-cz86b\") pod \"node-resolver-x4qcz\" (UID: \"c590b6a7-f5a5-476f-a0bd-fb8e8853b8cb\") " pod="openshift-dns/node-resolver-x4qcz" Apr 16 13:57:11.659203 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.659072 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cab34005-67cd-4db7-80d0-1d5b72338051-agent-certs\") pod \"konnectivity-agent-sflxw\" (UID: \"cab34005-67cd-4db7-80d0-1d5b72338051\") " pod="kube-system/konnectivity-agent-sflxw" Apr 16 13:57:11.659203 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.659107 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b83df07-d346-4caa-b358-294994a3d89b-system-cni-dir\") pod \"multus-8js6c\" (UID: \"5b83df07-d346-4caa-b358-294994a3d89b\") " pod="openshift-multus/multus-8js6c" Apr 16 13:57:11.659203 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.659150 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5b83df07-d346-4caa-b358-294994a3d89b-cni-binary-copy\") pod \"multus-8js6c\" (UID: \"5b83df07-d346-4caa-b358-294994a3d89b\") " pod="openshift-multus/multus-8js6c" Apr 16 13:57:11.659203 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.659174 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c18aa469-e518-4e2b-9d80-85e7e1d8ed94-run-ovn\") pod \"ovnkube-node-99w5m\" (UID: \"c18aa469-e518-4e2b-9d80-85e7e1d8ed94\") " pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" Apr 16 13:57:11.659203 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.659196 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c18aa469-e518-4e2b-9d80-85e7e1d8ed94-node-log\") pod \"ovnkube-node-99w5m\" (UID: \"c18aa469-e518-4e2b-9d80-85e7e1d8ed94\") " pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" Apr 16 13:57:11.659591 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.659218 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j57vl\" (UniqueName: \"kubernetes.io/projected/d29c2bfc-de60-442e-a4cb-615b6688e211-kube-api-access-j57vl\") pod \"aws-ebs-csi-driver-node-27wm7\" (UID: \"d29c2bfc-de60-442e-a4cb-615b6688e211\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27wm7" Apr 16 13:57:11.659591 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.659257 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ce951cb8-753a-4902-be96-172949512289-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rnvkv\" (UID: \"ce951cb8-753a-4902-be96-172949512289\") " pod="openshift-multus/multus-additional-cni-plugins-rnvkv" Apr 16 13:57:11.659591 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.659288 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5b83df07-d346-4caa-b358-294994a3d89b-multus-socket-dir-parent\") pod \"multus-8js6c\" (UID: \"5b83df07-d346-4caa-b358-294994a3d89b\") " pod="openshift-multus/multus-8js6c" Apr 16 13:57:11.659591 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.659306 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5b83df07-d346-4caa-b358-294994a3d89b-hostroot\") pod \"multus-8js6c\" (UID: \"5b83df07-d346-4caa-b358-294994a3d89b\") " pod="openshift-multus/multus-8js6c" Apr 16 13:57:11.659591 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.659321 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5b83df07-d346-4caa-b358-294994a3d89b-multus-conf-dir\") pod \"multus-8js6c\" (UID: \"5b83df07-d346-4caa-b358-294994a3d89b\") " pod="openshift-multus/multus-8js6c" Apr 16 13:57:11.659591 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.659336 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c18aa469-e518-4e2b-9d80-85e7e1d8ed94-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-99w5m\" (UID: \"c18aa469-e518-4e2b-9d80-85e7e1d8ed94\") " pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" Apr 16 13:57:11.659591 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.659365 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bbd94c79-236c-4a37-b6ad-384d38491b9e-etc-sysctl-d\") pod \"tuned-6srr9\" (UID: \"bbd94c79-236c-4a37-b6ad-384d38491b9e\") " pod="openshift-cluster-node-tuning-operator/tuned-6srr9" Apr 16 13:57:11.659591 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.659391 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ce951cb8-753a-4902-be96-172949512289-os-release\") pod \"multus-additional-cni-plugins-rnvkv\" (UID: \"ce951cb8-753a-4902-be96-172949512289\") " pod="openshift-multus/multus-additional-cni-plugins-rnvkv" Apr 16 13:57:11.659591 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.659406 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ce951cb8-753a-4902-be96-172949512289-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rnvkv\" (UID: \"ce951cb8-753a-4902-be96-172949512289\") " pod="openshift-multus/multus-additional-cni-plugins-rnvkv" Apr 16 13:57:11.659591 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.659436 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7d2abdf5-4927-45c4-9386-af7dc0ed8d90-host-slash\") pod \"iptables-alerter-v2ntl\" (UID: \"7d2abdf5-4927-45c4-9386-af7dc0ed8d90\") " pod="openshift-network-operator/iptables-alerter-v2ntl" Apr 16 13:57:11.659591 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.659436 2574 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Apr 16 13:57:11.659591 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.659462 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5b83df07-d346-4caa-b358-294994a3d89b-os-release\") pod \"multus-8js6c\" (UID: \"5b83df07-d346-4caa-b358-294994a3d89b\") " pod="openshift-multus/multus-8js6c" Apr 16 13:57:11.659591 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.659490 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c18aa469-e518-4e2b-9d80-85e7e1d8ed94-run-systemd\") pod \"ovnkube-node-99w5m\" (UID: \"c18aa469-e518-4e2b-9d80-85e7e1d8ed94\") " pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" Apr 16 13:57:11.659591 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.659513 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c18aa469-e518-4e2b-9d80-85e7e1d8ed94-ovnkube-config\") pod \"ovnkube-node-99w5m\" (UID: \"c18aa469-e518-4e2b-9d80-85e7e1d8ed94\") " pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" Apr 16 13:57:11.659591 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.659560 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t7wk\" (UniqueName: \"kubernetes.io/projected/c18aa469-e518-4e2b-9d80-85e7e1d8ed94-kube-api-access-9t7wk\") pod \"ovnkube-node-99w5m\" (UID: \"c18aa469-e518-4e2b-9d80-85e7e1d8ed94\") " pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" Apr 16 13:57:11.660178 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.659603 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d29c2bfc-de60-442e-a4cb-615b6688e211-socket-dir\") pod \"aws-ebs-csi-driver-node-27wm7\" (UID: \"d29c2bfc-de60-442e-a4cb-615b6688e211\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27wm7" Apr 16 13:57:11.660178 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.659631 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c590b6a7-f5a5-476f-a0bd-fb8e8853b8cb-hosts-file\") pod \"node-resolver-x4qcz\" (UID: \"c590b6a7-f5a5-476f-a0bd-fb8e8853b8cb\") " pod="openshift-dns/node-resolver-x4qcz" Apr 16 13:57:11.660178 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.659655 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bbd94c79-236c-4a37-b6ad-384d38491b9e-run\") pod \"tuned-6srr9\" (UID: \"bbd94c79-236c-4a37-b6ad-384d38491b9e\") " pod="openshift-cluster-node-tuning-operator/tuned-6srr9" Apr 16 13:57:11.660178 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.659678 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bbd94c79-236c-4a37-b6ad-384d38491b9e-etc-sysctl-conf\") pod \"tuned-6srr9\" (UID: \"bbd94c79-236c-4a37-b6ad-384d38491b9e\") " pod="openshift-cluster-node-tuning-operator/tuned-6srr9" Apr 16 13:57:11.660178 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.659699 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bbd94c79-236c-4a37-b6ad-384d38491b9e-host\") pod \"tuned-6srr9\" (UID: \"bbd94c79-236c-4a37-b6ad-384d38491b9e\") " pod="openshift-cluster-node-tuning-operator/tuned-6srr9" Apr 16 13:57:11.660178 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.659721 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b83df07-d346-4caa-b358-294994a3d89b-etc-kubernetes\") pod \"multus-8js6c\" (UID: \"5b83df07-d346-4caa-b358-294994a3d89b\") " pod="openshift-multus/multus-8js6c" Apr 16 13:57:11.660178 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.659767 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d29c2bfc-de60-442e-a4cb-615b6688e211-registration-dir\") pod \"aws-ebs-csi-driver-node-27wm7\" (UID: \"d29c2bfc-de60-442e-a4cb-615b6688e211\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27wm7" Apr 16 13:57:11.660178 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.659785 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ce951cb8-753a-4902-be96-172949512289-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rnvkv\" (UID: \"ce951cb8-753a-4902-be96-172949512289\") " pod="openshift-multus/multus-additional-cni-plugins-rnvkv" Apr 16 13:57:11.660178 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.659798 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c18aa469-e518-4e2b-9d80-85e7e1d8ed94-systemd-units\") pod \"ovnkube-node-99w5m\" (UID: \"c18aa469-e518-4e2b-9d80-85e7e1d8ed94\") " pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" Apr 16 13:57:11.660178 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.659813 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c18aa469-e518-4e2b-9d80-85e7e1d8ed94-host-cni-bin\") pod \"ovnkube-node-99w5m\" (UID: \"c18aa469-e518-4e2b-9d80-85e7e1d8ed94\") " pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" Apr 16 13:57:11.660178 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.659827 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzml7\" (UniqueName: \"kubernetes.io/projected/2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e-kube-api-access-bzml7\") pod \"network-metrics-daemon-5fh28\" (UID: \"2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e\") " pod="openshift-multus/network-metrics-daemon-5fh28" Apr 16 13:57:11.660178 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.659858 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bbd94c79-236c-4a37-b6ad-384d38491b9e-etc-modprobe-d\") pod \"tuned-6srr9\" (UID: \"bbd94c79-236c-4a37-b6ad-384d38491b9e\") " pod="openshift-cluster-node-tuning-operator/tuned-6srr9" Apr 16 13:57:11.660178 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.659882 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bbd94c79-236c-4a37-b6ad-384d38491b9e-etc-sysconfig\") pod \"tuned-6srr9\" (UID: \"bbd94c79-236c-4a37-b6ad-384d38491b9e\") " pod="openshift-cluster-node-tuning-operator/tuned-6srr9" Apr 16 13:57:11.660178 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.659896 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6l6c\" (UniqueName: \"kubernetes.io/projected/48194990-0de3-45a9-acc5-6fcc4c2760d0-kube-api-access-h6l6c\") pod \"node-ca-brqdj\" (UID: \"48194990-0de3-45a9-acc5-6fcc4c2760d0\") " pod="openshift-image-registry/node-ca-brqdj" Apr 16 13:57:11.660178 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.659910 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b83df07-d346-4caa-b358-294994a3d89b-host-run-netns\") pod \"multus-8js6c\" (UID: \"5b83df07-d346-4caa-b358-294994a3d89b\") " pod="openshift-multus/multus-8js6c" Apr 16 13:57:11.660178 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.659924 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c18aa469-e518-4e2b-9d80-85e7e1d8ed94-ovn-node-metrics-cert\") pod \"ovnkube-node-99w5m\" (UID: \"c18aa469-e518-4e2b-9d80-85e7e1d8ed94\") " pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" Apr 16 13:57:11.660645 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.659938 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c590b6a7-f5a5-476f-a0bd-fb8e8853b8cb-tmp-dir\") pod \"node-resolver-x4qcz\" (UID: \"c590b6a7-f5a5-476f-a0bd-fb8e8853b8cb\") " pod="openshift-dns/node-resolver-x4qcz" Apr 16 13:57:11.660645 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.659956 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ce951cb8-753a-4902-be96-172949512289-cni-binary-copy\") pod \"multus-additional-cni-plugins-rnvkv\" (UID: \"ce951cb8-753a-4902-be96-172949512289\") " pod="openshift-multus/multus-additional-cni-plugins-rnvkv" Apr 16 13:57:11.660645 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.659970 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcttw\" (UniqueName: \"kubernetes.io/projected/ce951cb8-753a-4902-be96-172949512289-kube-api-access-zcttw\") pod \"multus-additional-cni-plugins-rnvkv\" (UID: \"ce951cb8-753a-4902-be96-172949512289\") " pod="openshift-multus/multus-additional-cni-plugins-rnvkv" Apr 16 13:57:11.660645 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.659984 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c18aa469-e518-4e2b-9d80-85e7e1d8ed94-host-slash\") pod \"ovnkube-node-99w5m\" (UID: \"c18aa469-e518-4e2b-9d80-85e7e1d8ed94\") " pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" Apr 16 13:57:11.660645 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.659996 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c18aa469-e518-4e2b-9d80-85e7e1d8ed94-log-socket\") pod \"ovnkube-node-99w5m\" (UID: \"c18aa469-e518-4e2b-9d80-85e7e1d8ed94\") " pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" Apr 16 13:57:11.660645 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.660012 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d29c2bfc-de60-442e-a4cb-615b6688e211-device-dir\") pod \"aws-ebs-csi-driver-node-27wm7\" (UID: \"d29c2bfc-de60-442e-a4cb-615b6688e211\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27wm7" Apr 16 13:57:11.660645 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.660032 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhh6k\" (UniqueName: \"kubernetes.io/projected/df2f75cb-df70-4c82-b805-f93bcc7e8030-kube-api-access-lhh6k\") pod \"network-check-target-h4r6r\" (UID: \"df2f75cb-df70-4c82-b805-f93bcc7e8030\") " pod="openshift-network-diagnostics/network-check-target-h4r6r" Apr 16 13:57:11.660645 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.660057 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nv2v\" (UniqueName: \"kubernetes.io/projected/7d2abdf5-4927-45c4-9386-af7dc0ed8d90-kube-api-access-9nv2v\") pod \"iptables-alerter-v2ntl\" (UID: \"7d2abdf5-4927-45c4-9386-af7dc0ed8d90\") " pod="openshift-network-operator/iptables-alerter-v2ntl" Apr 16 13:57:11.660645 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.660083 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b83df07-d346-4caa-b358-294994a3d89b-host-var-lib-kubelet\") pod \"multus-8js6c\" (UID: \"5b83df07-d346-4caa-b358-294994a3d89b\") " pod="openshift-multus/multus-8js6c" Apr 16 13:57:11.660645 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.660102 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5b83df07-d346-4caa-b358-294994a3d89b-host-var-lib-cni-multus\") pod \"multus-8js6c\" (UID: \"5b83df07-d346-4caa-b358-294994a3d89b\") " pod="openshift-multus/multus-8js6c" Apr 16 13:57:11.660645 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.660115 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d29c2bfc-de60-442e-a4cb-615b6688e211-kubelet-dir\") pod \"aws-ebs-csi-driver-node-27wm7\" (UID: \"d29c2bfc-de60-442e-a4cb-615b6688e211\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27wm7" Apr 16 13:57:11.660645 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.660129 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cab34005-67cd-4db7-80d0-1d5b72338051-konnectivity-ca\") pod \"konnectivity-agent-sflxw\" (UID: \"cab34005-67cd-4db7-80d0-1d5b72338051\") " pod="kube-system/konnectivity-agent-sflxw" Apr 16 13:57:11.660645 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.660144 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bbd94c79-236c-4a37-b6ad-384d38491b9e-etc-kubernetes\") pod \"tuned-6srr9\" (UID: \"bbd94c79-236c-4a37-b6ad-384d38491b9e\") " pod="openshift-cluster-node-tuning-operator/tuned-6srr9" Apr 16 13:57:11.660645 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.660164 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bbd94c79-236c-4a37-b6ad-384d38491b9e-sys\") pod \"tuned-6srr9\" (UID: \"bbd94c79-236c-4a37-b6ad-384d38491b9e\") " pod="openshift-cluster-node-tuning-operator/tuned-6srr9" Apr 16 13:57:11.660645 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.660180 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5b83df07-d346-4caa-b358-294994a3d89b-cnibin\") pod \"multus-8js6c\" (UID: \"5b83df07-d346-4caa-b358-294994a3d89b\") " pod="openshift-multus/multus-8js6c" Apr 16 13:57:11.660645 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.660195 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c18aa469-e518-4e2b-9d80-85e7e1d8ed94-env-overrides\") pod \"ovnkube-node-99w5m\" (UID: \"c18aa469-e518-4e2b-9d80-85e7e1d8ed94\") " pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" Apr 16 13:57:11.661184 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.660209 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bbd94c79-236c-4a37-b6ad-384d38491b9e-etc-systemd\") pod \"tuned-6srr9\" (UID: \"bbd94c79-236c-4a37-b6ad-384d38491b9e\") " pod="openshift-cluster-node-tuning-operator/tuned-6srr9" Apr 16 13:57:11.661184 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.660222 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bbd94c79-236c-4a37-b6ad-384d38491b9e-lib-modules\") pod \"tuned-6srr9\" (UID: \"bbd94c79-236c-4a37-b6ad-384d38491b9e\") " pod="openshift-cluster-node-tuning-operator/tuned-6srr9" Apr 16 13:57:11.661184 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.660236 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bbd94c79-236c-4a37-b6ad-384d38491b9e-etc-tuned\") pod \"tuned-6srr9\" (UID: \"bbd94c79-236c-4a37-b6ad-384d38491b9e\") " pod="openshift-cluster-node-tuning-operator/tuned-6srr9" Apr 16 13:57:11.661184 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.660256 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bbd94c79-236c-4a37-b6ad-384d38491b9e-tmp\") pod \"tuned-6srr9\" (UID: \"bbd94c79-236c-4a37-b6ad-384d38491b9e\") " pod="openshift-cluster-node-tuning-operator/tuned-6srr9" Apr 16 13:57:11.661184 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.660270 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-725db\" (UniqueName: \"kubernetes.io/projected/5b83df07-d346-4caa-b358-294994a3d89b-kube-api-access-725db\") pod \"multus-8js6c\" (UID: \"5b83df07-d346-4caa-b358-294994a3d89b\") " pod="openshift-multus/multus-8js6c" Apr 16 13:57:11.661184 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.660288 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c18aa469-e518-4e2b-9d80-85e7e1d8ed94-host-run-netns\") pod \"ovnkube-node-99w5m\" (UID: \"c18aa469-e518-4e2b-9d80-85e7e1d8ed94\") " pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" Apr 16 13:57:11.661184 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.660314 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c18aa469-e518-4e2b-9d80-85e7e1d8ed94-host-run-ovn-kubernetes\") pod \"ovnkube-node-99w5m\" (UID: \"c18aa469-e518-4e2b-9d80-85e7e1d8ed94\") " pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" Apr 16 13:57:11.661184 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.660337 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ce951cb8-753a-4902-be96-172949512289-cnibin\") pod \"multus-additional-cni-plugins-rnvkv\" (UID: \"ce951cb8-753a-4902-be96-172949512289\") " pod="openshift-multus/multus-additional-cni-plugins-rnvkv" Apr 16 13:57:11.661184 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.660356 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5b83df07-d346-4caa-b358-294994a3d89b-host-run-k8s-cni-cncf-io\") pod \"multus-8js6c\" (UID: \"5b83df07-d346-4caa-b358-294994a3d89b\") " pod="openshift-multus/multus-8js6c" Apr 16 13:57:11.661184 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.660373 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c18aa469-e518-4e2b-9d80-85e7e1d8ed94-host-kubelet\") pod \"ovnkube-node-99w5m\" (UID: \"c18aa469-e518-4e2b-9d80-85e7e1d8ed94\") " pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" Apr 16 13:57:11.661184 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.660393 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ce951cb8-753a-4902-be96-172949512289-system-cni-dir\") pod \"multus-additional-cni-plugins-rnvkv\" (UID: \"ce951cb8-753a-4902-be96-172949512289\") " pod="openshift-multus/multus-additional-cni-plugins-rnvkv" Apr 16 13:57:11.661184 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.660410 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c18aa469-e518-4e2b-9d80-85e7e1d8ed94-ovnkube-script-lib\") pod \"ovnkube-node-99w5m\" (UID: \"c18aa469-e518-4e2b-9d80-85e7e1d8ed94\") " pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" Apr 16 13:57:11.681597 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.681581 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-67bbq" Apr 16 13:57:11.691505 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.691487 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-67bbq" Apr 16 13:57:11.761130 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.761110 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d29c2bfc-de60-442e-a4cb-615b6688e211-etc-selinux\") pod \"aws-ebs-csi-driver-node-27wm7\" (UID: \"d29c2bfc-de60-442e-a4cb-615b6688e211\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27wm7" Apr 16 13:57:11.761212 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.761139 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7d2abdf5-4927-45c4-9386-af7dc0ed8d90-iptables-alerter-script\") pod \"iptables-alerter-v2ntl\" (UID: \"7d2abdf5-4927-45c4-9386-af7dc0ed8d90\") " pod="openshift-network-operator/iptables-alerter-v2ntl" Apr 16 13:57:11.761212 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.761156 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/48194990-0de3-45a9-acc5-6fcc4c2760d0-host\") pod \"node-ca-brqdj\" (UID: \"48194990-0de3-45a9-acc5-6fcc4c2760d0\") " pod="openshift-image-registry/node-ca-brqdj" Apr 16 13:57:11.761212 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.761171 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5b83df07-d346-4caa-b358-294994a3d89b-host-run-multus-certs\") pod \"multus-8js6c\" (UID: \"5b83df07-d346-4caa-b358-294994a3d89b\") " pod="openshift-multus/multus-8js6c" Apr 16 13:57:11.761212 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.761188 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c18aa469-e518-4e2b-9d80-85e7e1d8ed94-var-lib-openvswitch\") pod \"ovnkube-node-99w5m\" (UID: \"c18aa469-e518-4e2b-9d80-85e7e1d8ed94\") " pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" Apr 16 13:57:11.761212 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.761202 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c18aa469-e518-4e2b-9d80-85e7e1d8ed94-host-cni-netd\") pod \"ovnkube-node-99w5m\" (UID: \"c18aa469-e518-4e2b-9d80-85e7e1d8ed94\") " pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" Apr 16 13:57:11.761417 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.761240 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c18aa469-e518-4e2b-9d80-85e7e1d8ed94-host-cni-netd\") pod \"ovnkube-node-99w5m\" (UID: \"c18aa469-e518-4e2b-9d80-85e7e1d8ed94\") " pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" Apr 16 13:57:11.761417 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.761239 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/48194990-0de3-45a9-acc5-6fcc4c2760d0-host\") pod \"node-ca-brqdj\" (UID: \"48194990-0de3-45a9-acc5-6fcc4c2760d0\") " pod="openshift-image-registry/node-ca-brqdj" Apr 16 13:57:11.761417 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.761248 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-cz86b\" (UniqueName: \"kubernetes.io/projected/c590b6a7-f5a5-476f-a0bd-fb8e8853b8cb-kube-api-access-cz86b\") pod \"node-resolver-x4qcz\" (UID: \"c590b6a7-f5a5-476f-a0bd-fb8e8853b8cb\") " pod="openshift-dns/node-resolver-x4qcz" Apr 16 13:57:11.761417 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.761269 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5b83df07-d346-4caa-b358-294994a3d89b-host-run-multus-certs\") pod \"multus-8js6c\" (UID: \"5b83df07-d346-4caa-b358-294994a3d89b\") " pod="openshift-multus/multus-8js6c" Apr 16 13:57:11.761417 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.761270 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d29c2bfc-de60-442e-a4cb-615b6688e211-etc-selinux\") pod \"aws-ebs-csi-driver-node-27wm7\" (UID: \"d29c2bfc-de60-442e-a4cb-615b6688e211\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27wm7" Apr 16 13:57:11.761417 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.761288 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cab34005-67cd-4db7-80d0-1d5b72338051-agent-certs\") pod \"konnectivity-agent-sflxw\" (UID: \"cab34005-67cd-4db7-80d0-1d5b72338051\") " pod="kube-system/konnectivity-agent-sflxw" Apr 16 13:57:11.761417 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.761302 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c18aa469-e518-4e2b-9d80-85e7e1d8ed94-var-lib-openvswitch\") pod \"ovnkube-node-99w5m\" (UID: \"c18aa469-e518-4e2b-9d80-85e7e1d8ed94\") " pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" Apr 16 13:57:11.761417 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.761356 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b83df07-d346-4caa-b358-294994a3d89b-system-cni-dir\") pod \"multus-8js6c\" (UID: \"5b83df07-d346-4caa-b358-294994a3d89b\") " pod="openshift-multus/multus-8js6c" Apr 16 13:57:11.761417 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.761385 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5b83df07-d346-4caa-b358-294994a3d89b-cni-binary-copy\") pod \"multus-8js6c\" (UID: \"5b83df07-d346-4caa-b358-294994a3d89b\") " pod="openshift-multus/multus-8js6c" Apr 16 13:57:11.761417 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.761401 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c18aa469-e518-4e2b-9d80-85e7e1d8ed94-run-ovn\") pod \"ovnkube-node-99w5m\" (UID: \"c18aa469-e518-4e2b-9d80-85e7e1d8ed94\") " pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" Apr 16 13:57:11.761417 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.761417 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c18aa469-e518-4e2b-9d80-85e7e1d8ed94-node-log\") pod \"ovnkube-node-99w5m\" (UID: \"c18aa469-e518-4e2b-9d80-85e7e1d8ed94\") " pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" Apr 16 13:57:11.761958 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.761436 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-j57vl\" (UniqueName: \"kubernetes.io/projected/d29c2bfc-de60-442e-a4cb-615b6688e211-kube-api-access-j57vl\") pod \"aws-ebs-csi-driver-node-27wm7\" (UID: \"d29c2bfc-de60-442e-a4cb-615b6688e211\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27wm7" Apr 16 13:57:11.761958 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.761483 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c18aa469-e518-4e2b-9d80-85e7e1d8ed94-node-log\") pod \"ovnkube-node-99w5m\" (UID: \"c18aa469-e518-4e2b-9d80-85e7e1d8ed94\") " pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" Apr 16 13:57:11.761958 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.761511 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c18aa469-e518-4e2b-9d80-85e7e1d8ed94-run-ovn\") pod \"ovnkube-node-99w5m\" (UID: \"c18aa469-e518-4e2b-9d80-85e7e1d8ed94\") " pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" Apr 16 13:57:11.761958 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.761581 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b83df07-d346-4caa-b358-294994a3d89b-system-cni-dir\") pod \"multus-8js6c\" (UID: \"5b83df07-d346-4caa-b358-294994a3d89b\") " pod="openshift-multus/multus-8js6c" Apr 16 13:57:11.761958 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.761591 2574 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Apr 16 13:57:11.761958 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.761626 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ce951cb8-753a-4902-be96-172949512289-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rnvkv\" (UID: \"ce951cb8-753a-4902-be96-172949512289\") " pod="openshift-multus/multus-additional-cni-plugins-rnvkv" Apr 16 13:57:11.761958 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.761658 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5b83df07-d346-4caa-b358-294994a3d89b-multus-socket-dir-parent\") pod \"multus-8js6c\" (UID: \"5b83df07-d346-4caa-b358-294994a3d89b\") " pod="openshift-multus/multus-8js6c" Apr 16 13:57:11.761958 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.761682 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5b83df07-d346-4caa-b358-294994a3d89b-hostroot\") pod \"multus-8js6c\" (UID: \"5b83df07-d346-4caa-b358-294994a3d89b\") " pod="openshift-multus/multus-8js6c" Apr 16 13:57:11.761958 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.761707 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5b83df07-d346-4caa-b358-294994a3d89b-multus-conf-dir\") pod \"multus-8js6c\" (UID: \"5b83df07-d346-4caa-b358-294994a3d89b\") " pod="openshift-multus/multus-8js6c" Apr 16 13:57:11.761958 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.761733 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c18aa469-e518-4e2b-9d80-85e7e1d8ed94-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-99w5m\" (UID: \"c18aa469-e518-4e2b-9d80-85e7e1d8ed94\") " pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" Apr 16 13:57:11.761958 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.761771 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5b83df07-d346-4caa-b358-294994a3d89b-multus-socket-dir-parent\") pod \"multus-8js6c\" (UID: \"5b83df07-d346-4caa-b358-294994a3d89b\") " pod="openshift-multus/multus-8js6c" Apr 16 13:57:11.761958 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.761786 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5b83df07-d346-4caa-b358-294994a3d89b-hostroot\") pod \"multus-8js6c\" (UID: \"5b83df07-d346-4caa-b358-294994a3d89b\") " pod="openshift-multus/multus-8js6c" Apr 16 13:57:11.761958 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.761777 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bbd94c79-236c-4a37-b6ad-384d38491b9e-etc-sysctl-d\") pod \"tuned-6srr9\" (UID: \"bbd94c79-236c-4a37-b6ad-384d38491b9e\") " pod="openshift-cluster-node-tuning-operator/tuned-6srr9" Apr 16 13:57:11.761958 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.761813 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5b83df07-d346-4caa-b358-294994a3d89b-multus-conf-dir\") pod \"multus-8js6c\" (UID: \"5b83df07-d346-4caa-b358-294994a3d89b\") " pod="openshift-multus/multus-8js6c" Apr 16 13:57:11.761958 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.761818 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c18aa469-e518-4e2b-9d80-85e7e1d8ed94-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-99w5m\" (UID: \"c18aa469-e518-4e2b-9d80-85e7e1d8ed94\") " pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" Apr 16 13:57:11.761958 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.761849 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ce951cb8-753a-4902-be96-172949512289-os-release\") pod \"multus-additional-cni-plugins-rnvkv\" (UID: \"ce951cb8-753a-4902-be96-172949512289\") " pod="openshift-multus/multus-additional-cni-plugins-rnvkv" Apr 16 13:57:11.761958 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.761875 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ce951cb8-753a-4902-be96-172949512289-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rnvkv\" (UID: \"ce951cb8-753a-4902-be96-172949512289\") " pod="openshift-multus/multus-additional-cni-plugins-rnvkv" Apr 16 13:57:11.761958 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.761899 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7d2abdf5-4927-45c4-9386-af7dc0ed8d90-host-slash\") pod \"iptables-alerter-v2ntl\" (UID: \"7d2abdf5-4927-45c4-9386-af7dc0ed8d90\") " pod="openshift-network-operator/iptables-alerter-v2ntl" Apr 16 13:57:11.762612 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.761904 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bbd94c79-236c-4a37-b6ad-384d38491b9e-etc-sysctl-d\") pod \"tuned-6srr9\" (UID: \"bbd94c79-236c-4a37-b6ad-384d38491b9e\") " pod="openshift-cluster-node-tuning-operator/tuned-6srr9" Apr 16 13:57:11.762612 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.761930 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5b83df07-d346-4caa-b358-294994a3d89b-os-release\") pod \"multus-8js6c\" (UID: \"5b83df07-d346-4caa-b358-294994a3d89b\") " pod="openshift-multus/multus-8js6c" Apr 16 13:57:11.762612 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.761940 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7d2abdf5-4927-45c4-9386-af7dc0ed8d90-host-slash\") pod \"iptables-alerter-v2ntl\" (UID: \"7d2abdf5-4927-45c4-9386-af7dc0ed8d90\") " pod="openshift-network-operator/iptables-alerter-v2ntl" Apr 16 13:57:11.762612 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.761956 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c18aa469-e518-4e2b-9d80-85e7e1d8ed94-run-systemd\") pod \"ovnkube-node-99w5m\" (UID: \"c18aa469-e518-4e2b-9d80-85e7e1d8ed94\") " pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" Apr 16 13:57:11.762612 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.761978 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c18aa469-e518-4e2b-9d80-85e7e1d8ed94-ovnkube-config\") pod \"ovnkube-node-99w5m\" (UID: \"c18aa469-e518-4e2b-9d80-85e7e1d8ed94\") " pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" Apr 16 13:57:11.762612 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.761998 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ce951cb8-753a-4902-be96-172949512289-os-release\") pod \"multus-additional-cni-plugins-rnvkv\" (UID: \"ce951cb8-753a-4902-be96-172949512289\") " pod="openshift-multus/multus-additional-cni-plugins-rnvkv" Apr 16 13:57:11.762612 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.762003 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9t7wk\" (UniqueName: \"kubernetes.io/projected/c18aa469-e518-4e2b-9d80-85e7e1d8ed94-kube-api-access-9t7wk\") pod \"ovnkube-node-99w5m\" (UID: \"c18aa469-e518-4e2b-9d80-85e7e1d8ed94\") " pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" Apr 16 13:57:11.762612 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.762028 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d29c2bfc-de60-442e-a4cb-615b6688e211-socket-dir\") pod \"aws-ebs-csi-driver-node-27wm7\" (UID: \"d29c2bfc-de60-442e-a4cb-615b6688e211\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27wm7" Apr 16 13:57:11.762612 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.762054 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5b83df07-d346-4caa-b358-294994a3d89b-os-release\") pod \"multus-8js6c\" (UID: \"5b83df07-d346-4caa-b358-294994a3d89b\") " pod="openshift-multus/multus-8js6c" Apr 16 13:57:11.762612 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.762052 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c590b6a7-f5a5-476f-a0bd-fb8e8853b8cb-hosts-file\") pod \"node-resolver-x4qcz\" (UID: \"c590b6a7-f5a5-476f-a0bd-fb8e8853b8cb\") " pod="openshift-dns/node-resolver-x4qcz" Apr 16 13:57:11.762612 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.762112 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bbd94c79-236c-4a37-b6ad-384d38491b9e-run\") pod \"tuned-6srr9\" (UID: \"bbd94c79-236c-4a37-b6ad-384d38491b9e\") " pod="openshift-cluster-node-tuning-operator/tuned-6srr9" Apr 16 13:57:11.762612 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.762172 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ce951cb8-753a-4902-be96-172949512289-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rnvkv\" (UID: \"ce951cb8-753a-4902-be96-172949512289\") " pod="openshift-multus/multus-additional-cni-plugins-rnvkv" Apr 16 13:57:11.762612 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.762188 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bbd94c79-236c-4a37-b6ad-384d38491b9e-run\") pod \"tuned-6srr9\" (UID: \"bbd94c79-236c-4a37-b6ad-384d38491b9e\") " pod="openshift-cluster-node-tuning-operator/tuned-6srr9" Apr 16 13:57:11.762612 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.762229 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d29c2bfc-de60-442e-a4cb-615b6688e211-socket-dir\") pod \"aws-ebs-csi-driver-node-27wm7\" (UID: \"d29c2bfc-de60-442e-a4cb-615b6688e211\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27wm7" Apr 16 13:57:11.762612 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.762245 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c590b6a7-f5a5-476f-a0bd-fb8e8853b8cb-hosts-file\") pod \"node-resolver-x4qcz\" (UID: \"c590b6a7-f5a5-476f-a0bd-fb8e8853b8cb\") " pod="openshift-dns/node-resolver-x4qcz" Apr 16 13:57:11.762612 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.762260 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c18aa469-e518-4e2b-9d80-85e7e1d8ed94-run-systemd\") pod \"ovnkube-node-99w5m\" (UID: \"c18aa469-e518-4e2b-9d80-85e7e1d8ed94\") " pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" Apr 16 13:57:11.762612 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.762267 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bbd94c79-236c-4a37-b6ad-384d38491b9e-etc-sysctl-conf\") pod \"tuned-6srr9\" (UID: \"bbd94c79-236c-4a37-b6ad-384d38491b9e\") " pod="openshift-cluster-node-tuning-operator/tuned-6srr9" Apr 16 13:57:11.762612 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.762236 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7d2abdf5-4927-45c4-9386-af7dc0ed8d90-iptables-alerter-script\") pod \"iptables-alerter-v2ntl\" (UID: \"7d2abdf5-4927-45c4-9386-af7dc0ed8d90\") " pod="openshift-network-operator/iptables-alerter-v2ntl" Apr 16 13:57:11.763349 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.762297 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bbd94c79-236c-4a37-b6ad-384d38491b9e-host\") pod \"tuned-6srr9\" (UID: \"bbd94c79-236c-4a37-b6ad-384d38491b9e\") " pod="openshift-cluster-node-tuning-operator/tuned-6srr9" Apr 16 13:57:11.763349 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.762322 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b83df07-d346-4caa-b358-294994a3d89b-etc-kubernetes\") pod \"multus-8js6c\" (UID: \"5b83df07-d346-4caa-b358-294994a3d89b\") " pod="openshift-multus/multus-8js6c" Apr 16 13:57:11.763349 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.762325 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ce951cb8-753a-4902-be96-172949512289-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rnvkv\" (UID: \"ce951cb8-753a-4902-be96-172949512289\") " pod="openshift-multus/multus-additional-cni-plugins-rnvkv" Apr 16 13:57:11.763349 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.762346 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d29c2bfc-de60-442e-a4cb-615b6688e211-registration-dir\") pod \"aws-ebs-csi-driver-node-27wm7\" (UID: \"d29c2bfc-de60-442e-a4cb-615b6688e211\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27wm7" Apr 16 13:57:11.763349 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.762370 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bbd94c79-236c-4a37-b6ad-384d38491b9e-host\") pod \"tuned-6srr9\" (UID: \"bbd94c79-236c-4a37-b6ad-384d38491b9e\") " pod="openshift-cluster-node-tuning-operator/tuned-6srr9" Apr 16 13:57:11.763349 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.762379 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b83df07-d346-4caa-b358-294994a3d89b-etc-kubernetes\") pod \"multus-8js6c\" (UID: \"5b83df07-d346-4caa-b358-294994a3d89b\") " pod="openshift-multus/multus-8js6c" Apr 16 13:57:11.763349 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.762372 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ce951cb8-753a-4902-be96-172949512289-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rnvkv\" (UID: \"ce951cb8-753a-4902-be96-172949512289\") " pod="openshift-multus/multus-additional-cni-plugins-rnvkv" Apr 16 13:57:11.763349 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.762402 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bbd94c79-236c-4a37-b6ad-384d38491b9e-etc-sysctl-conf\") pod \"tuned-6srr9\" (UID: \"bbd94c79-236c-4a37-b6ad-384d38491b9e\") " pod="openshift-cluster-node-tuning-operator/tuned-6srr9" Apr 16 13:57:11.763349 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.762414 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c18aa469-e518-4e2b-9d80-85e7e1d8ed94-systemd-units\") pod \"ovnkube-node-99w5m\" (UID: \"c18aa469-e518-4e2b-9d80-85e7e1d8ed94\") " pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" Apr 16 13:57:11.763349 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.762425 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d29c2bfc-de60-442e-a4cb-615b6688e211-registration-dir\") pod \"aws-ebs-csi-driver-node-27wm7\" (UID: \"d29c2bfc-de60-442e-a4cb-615b6688e211\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27wm7" Apr 16 13:57:11.763349 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.762456 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c18aa469-e518-4e2b-9d80-85e7e1d8ed94-systemd-units\") pod \"ovnkube-node-99w5m\" (UID: \"c18aa469-e518-4e2b-9d80-85e7e1d8ed94\") " pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" Apr 16 13:57:11.763349 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.762509 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c18aa469-e518-4e2b-9d80-85e7e1d8ed94-host-cni-bin\") pod \"ovnkube-node-99w5m\" (UID: \"c18aa469-e518-4e2b-9d80-85e7e1d8ed94\") " pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" Apr 16 13:57:11.763349 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.762530 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5b83df07-d346-4caa-b358-294994a3d89b-cni-binary-copy\") pod \"multus-8js6c\" (UID: \"5b83df07-d346-4caa-b358-294994a3d89b\") " pod="openshift-multus/multus-8js6c" Apr 16 13:57:11.763349 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.762535 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bzml7\" (UniqueName: \"kubernetes.io/projected/2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e-kube-api-access-bzml7\") pod \"network-metrics-daemon-5fh28\" (UID: \"2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e\") " pod="openshift-multus/network-metrics-daemon-5fh28" Apr 16 13:57:11.763349 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.762560 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bbd94c79-236c-4a37-b6ad-384d38491b9e-etc-modprobe-d\") pod \"tuned-6srr9\" (UID: \"bbd94c79-236c-4a37-b6ad-384d38491b9e\") " pod="openshift-cluster-node-tuning-operator/tuned-6srr9" Apr 16 13:57:11.763349 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.762580 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bbd94c79-236c-4a37-b6ad-384d38491b9e-etc-sysconfig\") pod \"tuned-6srr9\" (UID: \"bbd94c79-236c-4a37-b6ad-384d38491b9e\") " pod="openshift-cluster-node-tuning-operator/tuned-6srr9" Apr 16 13:57:11.763349 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.762602 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-h6l6c\" (UniqueName: \"kubernetes.io/projected/48194990-0de3-45a9-acc5-6fcc4c2760d0-kube-api-access-h6l6c\") pod \"node-ca-brqdj\" (UID: \"48194990-0de3-45a9-acc5-6fcc4c2760d0\") " pod="openshift-image-registry/node-ca-brqdj" Apr 16 13:57:11.764122 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.762615 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c18aa469-e518-4e2b-9d80-85e7e1d8ed94-host-cni-bin\") pod \"ovnkube-node-99w5m\" (UID: \"c18aa469-e518-4e2b-9d80-85e7e1d8ed94\") " pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" Apr 16 13:57:11.764122 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.762629 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b83df07-d346-4caa-b358-294994a3d89b-host-run-netns\") pod \"multus-8js6c\" (UID: \"5b83df07-d346-4caa-b358-294994a3d89b\") " pod="openshift-multus/multus-8js6c" Apr 16 13:57:11.764122 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.762653 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c18aa469-e518-4e2b-9d80-85e7e1d8ed94-ovn-node-metrics-cert\") pod \"ovnkube-node-99w5m\" (UID: \"c18aa469-e518-4e2b-9d80-85e7e1d8ed94\") " pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" Apr 16 13:57:11.764122 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.762676 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c590b6a7-f5a5-476f-a0bd-fb8e8853b8cb-tmp-dir\") pod \"node-resolver-x4qcz\" (UID: \"c590b6a7-f5a5-476f-a0bd-fb8e8853b8cb\") " pod="openshift-dns/node-resolver-x4qcz" Apr 16 13:57:11.764122 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.762677 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bbd94c79-236c-4a37-b6ad-384d38491b9e-etc-sysconfig\") pod \"tuned-6srr9\" (UID: \"bbd94c79-236c-4a37-b6ad-384d38491b9e\") " pod="openshift-cluster-node-tuning-operator/tuned-6srr9" Apr 16 13:57:11.764122 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.762683 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bbd94c79-236c-4a37-b6ad-384d38491b9e-etc-modprobe-d\") pod \"tuned-6srr9\" (UID: \"bbd94c79-236c-4a37-b6ad-384d38491b9e\") " pod="openshift-cluster-node-tuning-operator/tuned-6srr9" Apr 16 13:57:11.764122 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.762695 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ce951cb8-753a-4902-be96-172949512289-cni-binary-copy\") pod \"multus-additional-cni-plugins-rnvkv\" (UID: \"ce951cb8-753a-4902-be96-172949512289\") " pod="openshift-multus/multus-additional-cni-plugins-rnvkv" Apr 16 13:57:11.764122 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.762712 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c18aa469-e518-4e2b-9d80-85e7e1d8ed94-ovnkube-config\") pod \"ovnkube-node-99w5m\" (UID: \"c18aa469-e518-4e2b-9d80-85e7e1d8ed94\") " pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" Apr 16 13:57:11.764122 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.762716 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zcttw\" (UniqueName: \"kubernetes.io/projected/ce951cb8-753a-4902-be96-172949512289-kube-api-access-zcttw\") pod \"multus-additional-cni-plugins-rnvkv\" (UID: \"ce951cb8-753a-4902-be96-172949512289\") " pod="openshift-multus/multus-additional-cni-plugins-rnvkv" Apr 16 13:57:11.764122 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.762725 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b83df07-d346-4caa-b358-294994a3d89b-host-run-netns\") pod \"multus-8js6c\" (UID: \"5b83df07-d346-4caa-b358-294994a3d89b\") " pod="openshift-multus/multus-8js6c" Apr 16 13:57:11.764122 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.762755 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c18aa469-e518-4e2b-9d80-85e7e1d8ed94-host-slash\") pod \"ovnkube-node-99w5m\" (UID: \"c18aa469-e518-4e2b-9d80-85e7e1d8ed94\") " pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" Apr 16 13:57:11.764122 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.762793 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c18aa469-e518-4e2b-9d80-85e7e1d8ed94-host-slash\") pod \"ovnkube-node-99w5m\" (UID: \"c18aa469-e518-4e2b-9d80-85e7e1d8ed94\") " pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" Apr 16 13:57:11.764122 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.762941 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ce951cb8-753a-4902-be96-172949512289-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rnvkv\" (UID: \"ce951cb8-753a-4902-be96-172949512289\") " pod="openshift-multus/multus-additional-cni-plugins-rnvkv" Apr 16 13:57:11.764122 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.762975 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c18aa469-e518-4e2b-9d80-85e7e1d8ed94-log-socket\") pod \"ovnkube-node-99w5m\" (UID: \"c18aa469-e518-4e2b-9d80-85e7e1d8ed94\") " pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" Apr 16 13:57:11.764122 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.763009 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d29c2bfc-de60-442e-a4cb-615b6688e211-device-dir\") pod \"aws-ebs-csi-driver-node-27wm7\" (UID: \"d29c2bfc-de60-442e-a4cb-615b6688e211\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27wm7" Apr 16 13:57:11.764122 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.763057 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lhh6k\" (UniqueName: \"kubernetes.io/projected/df2f75cb-df70-4c82-b805-f93bcc7e8030-kube-api-access-lhh6k\") pod \"network-check-target-h4r6r\" (UID: \"df2f75cb-df70-4c82-b805-f93bcc7e8030\") " pod="openshift-network-diagnostics/network-check-target-h4r6r" Apr 16 13:57:11.764122 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.763079 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c18aa469-e518-4e2b-9d80-85e7e1d8ed94-log-socket\") pod \"ovnkube-node-99w5m\" (UID: \"c18aa469-e518-4e2b-9d80-85e7e1d8ed94\") " pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" Apr 16 13:57:11.764938 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.763085 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9nv2v\" (UniqueName: \"kubernetes.io/projected/7d2abdf5-4927-45c4-9386-af7dc0ed8d90-kube-api-access-9nv2v\") pod \"iptables-alerter-v2ntl\" (UID: \"7d2abdf5-4927-45c4-9386-af7dc0ed8d90\") " pod="openshift-network-operator/iptables-alerter-v2ntl" Apr 16 13:57:11.764938 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.763114 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c590b6a7-f5a5-476f-a0bd-fb8e8853b8cb-tmp-dir\") pod \"node-resolver-x4qcz\" (UID: \"c590b6a7-f5a5-476f-a0bd-fb8e8853b8cb\") " pod="openshift-dns/node-resolver-x4qcz" Apr 16 13:57:11.764938 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.763164 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b83df07-d346-4caa-b358-294994a3d89b-host-var-lib-kubelet\") pod \"multus-8js6c\" (UID: \"5b83df07-d346-4caa-b358-294994a3d89b\") " pod="openshift-multus/multus-8js6c" Apr 16 13:57:11.764938 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.763172 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ce951cb8-753a-4902-be96-172949512289-cni-binary-copy\") pod \"multus-additional-cni-plugins-rnvkv\" (UID: \"ce951cb8-753a-4902-be96-172949512289\") " pod="openshift-multus/multus-additional-cni-plugins-rnvkv" Apr 16 13:57:11.764938 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.763221 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b83df07-d346-4caa-b358-294994a3d89b-host-var-lib-kubelet\") pod \"multus-8js6c\" (UID: \"5b83df07-d346-4caa-b358-294994a3d89b\") " pod="openshift-multus/multus-8js6c" Apr 16 13:57:11.764938 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.763234 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d29c2bfc-de60-442e-a4cb-615b6688e211-device-dir\") pod \"aws-ebs-csi-driver-node-27wm7\" (UID: \"d29c2bfc-de60-442e-a4cb-615b6688e211\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27wm7" Apr 16 13:57:11.764938 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.763269 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5b83df07-d346-4caa-b358-294994a3d89b-host-var-lib-cni-multus\") pod \"multus-8js6c\" (UID: \"5b83df07-d346-4caa-b358-294994a3d89b\") " pod="openshift-multus/multus-8js6c" Apr 16 13:57:11.764938 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.763296 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d29c2bfc-de60-442e-a4cb-615b6688e211-kubelet-dir\") pod \"aws-ebs-csi-driver-node-27wm7\" (UID: \"d29c2bfc-de60-442e-a4cb-615b6688e211\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27wm7" Apr 16 13:57:11.764938 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.763330 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cab34005-67cd-4db7-80d0-1d5b72338051-konnectivity-ca\") pod \"konnectivity-agent-sflxw\" (UID: \"cab34005-67cd-4db7-80d0-1d5b72338051\") " pod="kube-system/konnectivity-agent-sflxw" Apr 16 13:57:11.764938 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.763353 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bbd94c79-236c-4a37-b6ad-384d38491b9e-etc-kubernetes\") pod \"tuned-6srr9\" (UID: \"bbd94c79-236c-4a37-b6ad-384d38491b9e\") " pod="openshift-cluster-node-tuning-operator/tuned-6srr9" Apr 16 13:57:11.764938 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.763377 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bbd94c79-236c-4a37-b6ad-384d38491b9e-sys\") pod \"tuned-6srr9\" (UID: \"bbd94c79-236c-4a37-b6ad-384d38491b9e\") " pod="openshift-cluster-node-tuning-operator/tuned-6srr9" Apr 16 13:57:11.764938 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.763381 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5b83df07-d346-4caa-b358-294994a3d89b-host-var-lib-cni-multus\") pod \"multus-8js6c\" (UID: \"5b83df07-d346-4caa-b358-294994a3d89b\") " pod="openshift-multus/multus-8js6c" Apr 16 13:57:11.764938 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.763401 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5b83df07-d346-4caa-b358-294994a3d89b-cnibin\") pod \"multus-8js6c\" (UID: \"5b83df07-d346-4caa-b358-294994a3d89b\") " pod="openshift-multus/multus-8js6c" Apr 16 13:57:11.764938 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.763442 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bbd94c79-236c-4a37-b6ad-384d38491b9e-etc-kubernetes\") pod \"tuned-6srr9\" (UID: \"bbd94c79-236c-4a37-b6ad-384d38491b9e\") " pod="openshift-cluster-node-tuning-operator/tuned-6srr9" Apr 16 13:57:11.764938 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.763460 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c18aa469-e518-4e2b-9d80-85e7e1d8ed94-env-overrides\") pod \"ovnkube-node-99w5m\" (UID: \"c18aa469-e518-4e2b-9d80-85e7e1d8ed94\") " pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" Apr 16 13:57:11.764938 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.763484 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bbd94c79-236c-4a37-b6ad-384d38491b9e-etc-systemd\") pod \"tuned-6srr9\" (UID: \"bbd94c79-236c-4a37-b6ad-384d38491b9e\") " pod="openshift-cluster-node-tuning-operator/tuned-6srr9" Apr 16 13:57:11.764938 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.763492 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d29c2bfc-de60-442e-a4cb-615b6688e211-kubelet-dir\") pod \"aws-ebs-csi-driver-node-27wm7\" (UID: \"d29c2bfc-de60-442e-a4cb-615b6688e211\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27wm7" Apr 16 13:57:11.764938 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.763493 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bbd94c79-236c-4a37-b6ad-384d38491b9e-sys\") pod \"tuned-6srr9\" (UID: \"bbd94c79-236c-4a37-b6ad-384d38491b9e\") " pod="openshift-cluster-node-tuning-operator/tuned-6srr9" Apr 16 13:57:11.765766 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.763507 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bbd94c79-236c-4a37-b6ad-384d38491b9e-lib-modules\") pod \"tuned-6srr9\" (UID: \"bbd94c79-236c-4a37-b6ad-384d38491b9e\") " pod="openshift-cluster-node-tuning-operator/tuned-6srr9" Apr 16 13:57:11.765766 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.763531 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bbd94c79-236c-4a37-b6ad-384d38491b9e-etc-tuned\") pod \"tuned-6srr9\" (UID: \"bbd94c79-236c-4a37-b6ad-384d38491b9e\") " pod="openshift-cluster-node-tuning-operator/tuned-6srr9" Apr 16 13:57:11.765766 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.763551 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bbd94c79-236c-4a37-b6ad-384d38491b9e-tmp\") pod \"tuned-6srr9\" (UID: \"bbd94c79-236c-4a37-b6ad-384d38491b9e\") " pod="openshift-cluster-node-tuning-operator/tuned-6srr9" Apr 16 13:57:11.765766 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.763578 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-725db\" (UniqueName: \"kubernetes.io/projected/5b83df07-d346-4caa-b358-294994a3d89b-kube-api-access-725db\") pod \"multus-8js6c\" (UID: \"5b83df07-d346-4caa-b358-294994a3d89b\") " pod="openshift-multus/multus-8js6c" Apr 16 13:57:11.765766 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.763602 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c18aa469-e518-4e2b-9d80-85e7e1d8ed94-host-run-netns\") pod \"ovnkube-node-99w5m\" (UID: \"c18aa469-e518-4e2b-9d80-85e7e1d8ed94\") " pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" Apr 16 13:57:11.765766 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.763627 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c18aa469-e518-4e2b-9d80-85e7e1d8ed94-host-run-ovn-kubernetes\") pod \"ovnkube-node-99w5m\" (UID: \"c18aa469-e518-4e2b-9d80-85e7e1d8ed94\") " pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" Apr 16 13:57:11.765766 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.763632 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bbd94c79-236c-4a37-b6ad-384d38491b9e-lib-modules\") pod \"tuned-6srr9\" (UID: \"bbd94c79-236c-4a37-b6ad-384d38491b9e\") " pod="openshift-cluster-node-tuning-operator/tuned-6srr9" Apr 16 13:57:11.765766 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.763660 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ce951cb8-753a-4902-be96-172949512289-cnibin\") pod \"multus-additional-cni-plugins-rnvkv\" (UID: \"ce951cb8-753a-4902-be96-172949512289\") " pod="openshift-multus/multus-additional-cni-plugins-rnvkv" Apr 16 13:57:11.765766 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.763680 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bbd94c79-236c-4a37-b6ad-384d38491b9e-etc-systemd\") pod \"tuned-6srr9\" (UID: \"bbd94c79-236c-4a37-b6ad-384d38491b9e\") " pod="openshift-cluster-node-tuning-operator/tuned-6srr9" Apr 16 13:57:11.765766 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.763684 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5b83df07-d346-4caa-b358-294994a3d89b-host-run-k8s-cni-cncf-io\") pod \"multus-8js6c\" (UID: \"5b83df07-d346-4caa-b358-294994a3d89b\") " pod="openshift-multus/multus-8js6c" Apr 16 13:57:11.765766 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.763707 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c18aa469-e518-4e2b-9d80-85e7e1d8ed94-host-kubelet\") pod \"ovnkube-node-99w5m\" (UID: \"c18aa469-e518-4e2b-9d80-85e7e1d8ed94\") " pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" Apr 16 13:57:11.765766 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.763732 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ce951cb8-753a-4902-be96-172949512289-system-cni-dir\") pod \"multus-additional-cni-plugins-rnvkv\" (UID: \"ce951cb8-753a-4902-be96-172949512289\") " pod="openshift-multus/multus-additional-cni-plugins-rnvkv" Apr 16 13:57:11.765766 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.763771 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c18aa469-e518-4e2b-9d80-85e7e1d8ed94-ovnkube-script-lib\") pod \"ovnkube-node-99w5m\" (UID: \"c18aa469-e518-4e2b-9d80-85e7e1d8ed94\") " pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" Apr 16 13:57:11.765766 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.763798 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bbd94c79-236c-4a37-b6ad-384d38491b9e-var-lib-kubelet\") pod \"tuned-6srr9\" (UID: \"bbd94c79-236c-4a37-b6ad-384d38491b9e\") " pod="openshift-cluster-node-tuning-operator/tuned-6srr9" Apr 16 13:57:11.765766 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.763823 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/48194990-0de3-45a9-acc5-6fcc4c2760d0-serviceca\") pod \"node-ca-brqdj\" (UID: \"48194990-0de3-45a9-acc5-6fcc4c2760d0\") " pod="openshift-image-registry/node-ca-brqdj" Apr 16 13:57:11.765766 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.763875 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b83df07-d346-4caa-b358-294994a3d89b-multus-cni-dir\") pod \"multus-8js6c\" (UID: \"5b83df07-d346-4caa-b358-294994a3d89b\") " pod="openshift-multus/multus-8js6c" Apr 16 13:57:11.765766 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.763904 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c18aa469-e518-4e2b-9d80-85e7e1d8ed94-etc-openvswitch\") pod \"ovnkube-node-99w5m\" (UID: \"c18aa469-e518-4e2b-9d80-85e7e1d8ed94\") " pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" Apr 16 13:57:11.766335 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.763921 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c18aa469-e518-4e2b-9d80-85e7e1d8ed94-env-overrides\") pod \"ovnkube-node-99w5m\" (UID: \"c18aa469-e518-4e2b-9d80-85e7e1d8ed94\") " pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" Apr 16 13:57:11.766335 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.763929 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c18aa469-e518-4e2b-9d80-85e7e1d8ed94-run-openvswitch\") pod \"ovnkube-node-99w5m\" (UID: \"c18aa469-e518-4e2b-9d80-85e7e1d8ed94\") " pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" Apr 16 13:57:11.766335 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.763953 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c18aa469-e518-4e2b-9d80-85e7e1d8ed94-host-kubelet\") pod \"ovnkube-node-99w5m\" (UID: \"c18aa469-e518-4e2b-9d80-85e7e1d8ed94\") " pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" Apr 16 13:57:11.766335 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.763955 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d29c2bfc-de60-442e-a4cb-615b6688e211-sys-fs\") pod \"aws-ebs-csi-driver-node-27wm7\" (UID: \"d29c2bfc-de60-442e-a4cb-615b6688e211\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27wm7" Apr 16 13:57:11.766335 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.763999 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e-metrics-certs\") pod \"network-metrics-daemon-5fh28\" (UID: \"2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e\") " pod="openshift-multus/network-metrics-daemon-5fh28" Apr 16 13:57:11.766335 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.764028 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zjn79\" (UniqueName: \"kubernetes.io/projected/bbd94c79-236c-4a37-b6ad-384d38491b9e-kube-api-access-zjn79\") pod \"tuned-6srr9\" (UID: \"bbd94c79-236c-4a37-b6ad-384d38491b9e\") " pod="openshift-cluster-node-tuning-operator/tuned-6srr9" Apr 16 13:57:11.766335 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.764036 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c18aa469-e518-4e2b-9d80-85e7e1d8ed94-host-run-ovn-kubernetes\") pod \"ovnkube-node-99w5m\" (UID: \"c18aa469-e518-4e2b-9d80-85e7e1d8ed94\") " pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" Apr 16 13:57:11.766335 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.764054 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b83df07-d346-4caa-b358-294994a3d89b-host-var-lib-cni-bin\") pod \"multus-8js6c\" (UID: \"5b83df07-d346-4caa-b358-294994a3d89b\") " pod="openshift-multus/multus-8js6c" Apr 16 13:57:11.766335 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.764076 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ce951cb8-753a-4902-be96-172949512289-cnibin\") pod \"multus-additional-cni-plugins-rnvkv\" (UID: \"ce951cb8-753a-4902-be96-172949512289\") " pod="openshift-multus/multus-additional-cni-plugins-rnvkv" Apr 16 13:57:11.766335 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.764083 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5b83df07-d346-4caa-b358-294994a3d89b-multus-daemon-config\") pod \"multus-8js6c\" (UID: \"5b83df07-d346-4caa-b358-294994a3d89b\") " pod="openshift-multus/multus-8js6c" Apr 16 13:57:11.766335 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.764085 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cab34005-67cd-4db7-80d0-1d5b72338051-konnectivity-ca\") pod \"konnectivity-agent-sflxw\" (UID: \"cab34005-67cd-4db7-80d0-1d5b72338051\") " pod="kube-system/konnectivity-agent-sflxw" Apr 16 13:57:11.766335 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.764000 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c18aa469-e518-4e2b-9d80-85e7e1d8ed94-host-run-netns\") pod \"ovnkube-node-99w5m\" (UID: \"c18aa469-e518-4e2b-9d80-85e7e1d8ed94\") " pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" Apr 16 13:57:11.766335 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.764117 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5b83df07-d346-4caa-b358-294994a3d89b-host-run-k8s-cni-cncf-io\") pod \"multus-8js6c\" (UID: \"5b83df07-d346-4caa-b358-294994a3d89b\") " pod="openshift-multus/multus-8js6c" Apr 16 13:57:11.766335 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.764130 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c18aa469-e518-4e2b-9d80-85e7e1d8ed94-run-openvswitch\") pod \"ovnkube-node-99w5m\" (UID: \"c18aa469-e518-4e2b-9d80-85e7e1d8ed94\") " pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" Apr 16 13:57:11.766335 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.763553 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5b83df07-d346-4caa-b358-294994a3d89b-cnibin\") pod \"multus-8js6c\" (UID: \"5b83df07-d346-4caa-b358-294994a3d89b\") " pod="openshift-multus/multus-8js6c" Apr 16 13:57:11.766335 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.764179 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d29c2bfc-de60-442e-a4cb-615b6688e211-sys-fs\") pod \"aws-ebs-csi-driver-node-27wm7\" (UID: \"d29c2bfc-de60-442e-a4cb-615b6688e211\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27wm7" Apr 16 13:57:11.766335 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.764183 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ce951cb8-753a-4902-be96-172949512289-system-cni-dir\") pod \"multus-additional-cni-plugins-rnvkv\" (UID: \"ce951cb8-753a-4902-be96-172949512289\") " pod="openshift-multus/multus-additional-cni-plugins-rnvkv" Apr 16 13:57:11.766837 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.764227 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b83df07-d346-4caa-b358-294994a3d89b-host-var-lib-cni-bin\") pod \"multus-8js6c\" (UID: \"5b83df07-d346-4caa-b358-294994a3d89b\") " pod="openshift-multus/multus-8js6c" Apr 16 13:57:11.766837 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:11.764277 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Apr 16 13:57:11.766837 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:11.764512 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e-metrics-certs podName:2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e nodeName:}" failed. No retries permitted until 2026-04-16 13:57:12.264469602 +0000 UTC m=+2.094204467 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e-metrics-certs") pod "network-metrics-daemon-5fh28" (UID: "2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e") : object "openshift-multus"/"metrics-daemon-secret" not registered Apr 16 13:57:11.766837 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.764706 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bbd94c79-236c-4a37-b6ad-384d38491b9e-var-lib-kubelet\") pod \"tuned-6srr9\" (UID: \"bbd94c79-236c-4a37-b6ad-384d38491b9e\") " pod="openshift-cluster-node-tuning-operator/tuned-6srr9" Apr 16 13:57:11.766837 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.764737 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5b83df07-d346-4caa-b358-294994a3d89b-multus-daemon-config\") pod \"multus-8js6c\" (UID: \"5b83df07-d346-4caa-b358-294994a3d89b\") " pod="openshift-multus/multus-8js6c" Apr 16 13:57:11.766837 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.764790 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c18aa469-e518-4e2b-9d80-85e7e1d8ed94-etc-openvswitch\") pod \"ovnkube-node-99w5m\" (UID: \"c18aa469-e518-4e2b-9d80-85e7e1d8ed94\") " pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" Apr 16 13:57:11.766837 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.764802 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b83df07-d346-4caa-b358-294994a3d89b-multus-cni-dir\") pod \"multus-8js6c\" (UID: \"5b83df07-d346-4caa-b358-294994a3d89b\") " pod="openshift-multus/multus-8js6c" Apr 16 13:57:11.766837 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.764877 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/48194990-0de3-45a9-acc5-6fcc4c2760d0-serviceca\") pod \"node-ca-brqdj\" (UID: \"48194990-0de3-45a9-acc5-6fcc4c2760d0\") " pod="openshift-image-registry/node-ca-brqdj" Apr 16 13:57:11.766837 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.765136 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c18aa469-e518-4e2b-9d80-85e7e1d8ed94-ovn-node-metrics-cert\") pod \"ovnkube-node-99w5m\" (UID: \"c18aa469-e518-4e2b-9d80-85e7e1d8ed94\") " pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" Apr 16 13:57:11.766837 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.765230 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cab34005-67cd-4db7-80d0-1d5b72338051-agent-certs\") pod \"konnectivity-agent-sflxw\" (UID: \"cab34005-67cd-4db7-80d0-1d5b72338051\") " pod="kube-system/konnectivity-agent-sflxw" Apr 16 13:57:11.766837 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.765441 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c18aa469-e518-4e2b-9d80-85e7e1d8ed94-ovnkube-script-lib\") pod \"ovnkube-node-99w5m\" (UID: \"c18aa469-e518-4e2b-9d80-85e7e1d8ed94\") " pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" Apr 16 13:57:11.766837 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.766185 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bbd94c79-236c-4a37-b6ad-384d38491b9e-etc-tuned\") pod \"tuned-6srr9\" (UID: \"bbd94c79-236c-4a37-b6ad-384d38491b9e\") " pod="openshift-cluster-node-tuning-operator/tuned-6srr9" Apr 16 13:57:11.766837 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.766417 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bbd94c79-236c-4a37-b6ad-384d38491b9e-tmp\") pod \"tuned-6srr9\" (UID: \"bbd94c79-236c-4a37-b6ad-384d38491b9e\") " pod="openshift-cluster-node-tuning-operator/tuned-6srr9" Apr 16 13:57:11.769127 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:11.769106 2574 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Apr 16 13:57:11.769127 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:11.769129 2574 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Apr 16 13:57:11.769127 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:11.769138 2574 projected.go:194] Error preparing data for projected volume kube-api-access-lhh6k for pod openshift-network-diagnostics/network-check-target-h4r6r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 16 13:57:11.769294 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:11.769203 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df2f75cb-df70-4c82-b805-f93bcc7e8030-kube-api-access-lhh6k podName:df2f75cb-df70-4c82-b805-f93bcc7e8030 nodeName:}" failed. No retries permitted until 2026-04-16 13:57:12.269191532 +0000 UTC m=+2.098926383 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-lhh6k" (UniqueName: "kubernetes.io/projected/df2f75cb-df70-4c82-b805-f93bcc7e8030-kube-api-access-lhh6k") pod "network-check-target-h4r6r" (UID: "df2f75cb-df70-4c82-b805-f93bcc7e8030") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 16 13:57:11.771575 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.771167 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t7wk\" (UniqueName: \"kubernetes.io/projected/c18aa469-e518-4e2b-9d80-85e7e1d8ed94-kube-api-access-9t7wk\") pod \"ovnkube-node-99w5m\" (UID: \"c18aa469-e518-4e2b-9d80-85e7e1d8ed94\") " pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" Apr 16 13:57:11.771575 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.771530 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nv2v\" (UniqueName: \"kubernetes.io/projected/7d2abdf5-4927-45c4-9386-af7dc0ed8d90-kube-api-access-9nv2v\") pod \"iptables-alerter-v2ntl\" (UID: \"7d2abdf5-4927-45c4-9386-af7dc0ed8d90\") " pod="openshift-network-operator/iptables-alerter-v2ntl" Apr 16 13:57:11.772008 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.771980 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzml7\" (UniqueName: \"kubernetes.io/projected/2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e-kube-api-access-bzml7\") pod \"network-metrics-daemon-5fh28\" (UID: \"2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e\") " pod="openshift-multus/network-metrics-daemon-5fh28" Apr 16 13:57:11.772137 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.772117 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j57vl\" (UniqueName: \"kubernetes.io/projected/d29c2bfc-de60-442e-a4cb-615b6688e211-kube-api-access-j57vl\") pod \"aws-ebs-csi-driver-node-27wm7\" (UID: \"d29c2bfc-de60-442e-a4cb-615b6688e211\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27wm7" Apr 16 13:57:11.772225 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.772170 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz86b\" (UniqueName: \"kubernetes.io/projected/c590b6a7-f5a5-476f-a0bd-fb8e8853b8cb-kube-api-access-cz86b\") pod \"node-resolver-x4qcz\" (UID: \"c590b6a7-f5a5-476f-a0bd-fb8e8853b8cb\") " pod="openshift-dns/node-resolver-x4qcz" Apr 16 13:57:11.772279 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.772241 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6l6c\" (UniqueName: \"kubernetes.io/projected/48194990-0de3-45a9-acc5-6fcc4c2760d0-kube-api-access-h6l6c\") pod \"node-ca-brqdj\" (UID: \"48194990-0de3-45a9-acc5-6fcc4c2760d0\") " pod="openshift-image-registry/node-ca-brqdj" Apr 16 13:57:11.772971 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.772950 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcttw\" (UniqueName: \"kubernetes.io/projected/ce951cb8-753a-4902-be96-172949512289-kube-api-access-zcttw\") pod \"multus-additional-cni-plugins-rnvkv\" (UID: \"ce951cb8-753a-4902-be96-172949512289\") " pod="openshift-multus/multus-additional-cni-plugins-rnvkv" Apr 16 13:57:11.773654 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.773638 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjn79\" (UniqueName: \"kubernetes.io/projected/bbd94c79-236c-4a37-b6ad-384d38491b9e-kube-api-access-zjn79\") pod \"tuned-6srr9\" (UID: \"bbd94c79-236c-4a37-b6ad-384d38491b9e\") " pod="openshift-cluster-node-tuning-operator/tuned-6srr9" Apr 16 13:57:11.774060 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.774042 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-725db\" (UniqueName: \"kubernetes.io/projected/5b83df07-d346-4caa-b358-294994a3d89b-kube-api-access-725db\") pod \"multus-8js6c\" (UID: \"5b83df07-d346-4caa-b358-294994a3d89b\") " pod="openshift-multus/multus-8js6c" Apr 16 13:57:11.786632 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.786612 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-brqdj" Apr 16 13:57:11.791692 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.791676 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8js6c" Apr 16 13:57:11.964899 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.964864 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" Apr 16 13:57:11.975533 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:11.975506 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sflxw" Apr 16 13:57:12.000133 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:12.000114 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27wm7" Apr 16 13:57:12.006126 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:12.006096 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcab34005_67cd_4db7_80d0_1d5b72338051.slice/crio-844a27c63895e1aafe7ec32d18dc33c2f1c01c80e6d1f6d9b3deda1e507b5979 WatchSource:0}: Error finding container 844a27c63895e1aafe7ec32d18dc33c2f1c01c80e6d1f6d9b3deda1e507b5979: Status 404 returned error can't find the container with id 844a27c63895e1aafe7ec32d18dc33c2f1c01c80e6d1f6d9b3deda1e507b5979 Apr 16 13:57:12.006571 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:12.006552 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod031b74a3d41a1b92a3331318a0772b2d.slice/crio-9e6e62a1a9db073f534c0c5734cbedc6efe6965fbefc8c1a6836fa151f75f9b2 WatchSource:0}: Error finding container 9e6e62a1a9db073f534c0c5734cbedc6efe6965fbefc8c1a6836fa151f75f9b2: Status 404 returned error can't find the container with id 9e6e62a1a9db073f534c0c5734cbedc6efe6965fbefc8c1a6836fa151f75f9b2 Apr 16 13:57:12.007539 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:12.007522 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd29c2bfc_de60_442e_a4cb_615b6688e211.slice/crio-ce54622af3fa106e7e5d84f805ee99d56033b052456554bced0c3465b328ef39 WatchSource:0}: Error finding container ce54622af3fa106e7e5d84f805ee99d56033b052456554bced0c3465b328ef39: Status 404 returned error can't find the container with id ce54622af3fa106e7e5d84f805ee99d56033b052456554bced0c3465b328ef39 Apr 16 13:57:12.012687 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:12.012670 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Apr 16 13:57:12.014318 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:12.014302 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6srr9" Apr 16 13:57:12.017849 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:12.017830 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rnvkv" Apr 16 13:57:12.020361 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:12.020344 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbd94c79_236c_4a37_b6ad_384d38491b9e.slice/crio-90b9d080cc1c2a76acb118dc1b777e2bcc613312160e5060c796c287f354da30 WatchSource:0}: Error finding container 90b9d080cc1c2a76acb118dc1b777e2bcc613312160e5060c796c287f354da30: Status 404 returned error can't find the container with id 90b9d080cc1c2a76acb118dc1b777e2bcc613312160e5060c796c287f354da30 Apr 16 13:57:12.025197 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:12.025179 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce951cb8_753a_4902_be96_172949512289.slice/crio-282f364a69d034fcf0dad8dd7fd660e0d76c9183057601d270dcc09507a087dd WatchSource:0}: Error finding container 282f364a69d034fcf0dad8dd7fd660e0d76c9183057601d270dcc09507a087dd: Status 404 returned error can't find the container with id 282f364a69d034fcf0dad8dd7fd660e0d76c9183057601d270dcc09507a087dd Apr 16 13:57:12.035792 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:12.035777 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-v2ntl" Apr 16 13:57:12.042950 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:12.042934 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d2abdf5_4927_45c4_9386_af7dc0ed8d90.slice/crio-54d111f496b9fea150690fdd64ff9f10239b53093e5073d635b218b5df7a85af WatchSource:0}: Error finding container 54d111f496b9fea150690fdd64ff9f10239b53093e5073d635b218b5df7a85af: Status 404 returned error can't find the container with id 54d111f496b9fea150690fdd64ff9f10239b53093e5073d635b218b5df7a85af Apr 16 13:57:12.057467 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:12.057443 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-x4qcz" Apr 16 13:57:12.062447 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:12.062423 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc590b6a7_f5a5_476f_a0bd_fb8e8853b8cb.slice/crio-5da39879e6c82da98970a57ff3eb2bea177b4b0c2c283146af70151b3d6aa604 WatchSource:0}: Error finding container 5da39879e6c82da98970a57ff3eb2bea177b4b0c2c283146af70151b3d6aa604: Status 404 returned error can't find the container with id 5da39879e6c82da98970a57ff3eb2bea177b4b0c2c283146af70151b3d6aa604 Apr 16 13:57:12.215043 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:12.215011 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48194990_0de3_45a9_acc5_6fcc4c2760d0.slice/crio-7561c9c68530a9f6f85ac9c37643bcbbb807aac466200239a10f4f923e67fd3e WatchSource:0}: Error finding container 7561c9c68530a9f6f85ac9c37643bcbbb807aac466200239a10f4f923e67fd3e: Status 404 returned error can't find the container with id 7561c9c68530a9f6f85ac9c37643bcbbb807aac466200239a10f4f923e67fd3e Apr 16 13:57:12.266985 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:12.266954 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e-metrics-certs\") pod \"network-metrics-daemon-5fh28\" (UID: \"2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e\") " pod="openshift-multus/network-metrics-daemon-5fh28" Apr 16 13:57:12.267159 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:12.267061 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Apr 16 13:57:12.267159 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:12.267119 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e-metrics-certs podName:2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e nodeName:}" failed. No retries permitted until 2026-04-16 13:57:13.26710535 +0000 UTC m=+3.096840199 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e-metrics-certs") pod "network-metrics-daemon-5fh28" (UID: "2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e") : object "openshift-multus"/"metrics-daemon-secret" not registered Apr 16 13:57:12.367706 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:12.367673 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lhh6k\" (UniqueName: \"kubernetes.io/projected/df2f75cb-df70-4c82-b805-f93bcc7e8030-kube-api-access-lhh6k\") pod \"network-check-target-h4r6r\" (UID: \"df2f75cb-df70-4c82-b805-f93bcc7e8030\") " pod="openshift-network-diagnostics/network-check-target-h4r6r" Apr 16 13:57:12.367870 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:12.367808 2574 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Apr 16 13:57:12.367870 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:12.367862 2574 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Apr 16 13:57:12.367957 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:12.367873 2574 projected.go:194] Error preparing data for projected volume kube-api-access-lhh6k for pod openshift-network-diagnostics/network-check-target-h4r6r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 16 13:57:12.367957 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:12.367944 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df2f75cb-df70-4c82-b805-f93bcc7e8030-kube-api-access-lhh6k podName:df2f75cb-df70-4c82-b805-f93bcc7e8030 nodeName:}" failed. No retries permitted until 2026-04-16 13:57:13.367924286 +0000 UTC m=+3.197659134 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-lhh6k" (UniqueName: "kubernetes.io/projected/df2f75cb-df70-4c82-b805-f93bcc7e8030-kube-api-access-lhh6k") pod "network-check-target-h4r6r" (UID: "df2f75cb-df70-4c82-b805-f93bcc7e8030") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 16 13:57:12.522187 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:12.522104 2574 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Apr 16 13:57:12.581182 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:12.581158 2574 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Apr 16 13:57:12.693870 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:12.693830 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-04-15 13:52:11 +0000 UTC" deadline="2027-11-05 10:43:18.895772143 +0000 UTC" Apr 16 13:57:12.693870 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:12.693867 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13628h46m6.201909088s" Apr 16 13:57:12.805982 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:12.805915 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5fh28" Apr 16 13:57:12.806124 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:12.806043 2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5fh28" podUID="2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e" Apr 16 13:57:12.806572 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:12.806550 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4r6r" Apr 16 13:57:12.806670 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:12.806653 2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4r6r" podUID="df2f75cb-df70-4c82-b805-f93bcc7e8030" Apr 16 13:57:12.812981 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:12.812933 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-x4qcz" event={"ID":"c590b6a7-f5a5-476f-a0bd-fb8e8853b8cb","Type":"ContainerStarted","Data":"5da39879e6c82da98970a57ff3eb2bea177b4b0c2c283146af70151b3d6aa604"} Apr 16 13:57:12.816622 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:12.816597 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-v2ntl" event={"ID":"7d2abdf5-4927-45c4-9386-af7dc0ed8d90","Type":"ContainerStarted","Data":"54d111f496b9fea150690fdd64ff9f10239b53093e5073d635b218b5df7a85af"} Apr 16 13:57:12.819474 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:12.819452 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rnvkv" event={"ID":"ce951cb8-753a-4902-be96-172949512289","Type":"ContainerStarted","Data":"282f364a69d034fcf0dad8dd7fd660e0d76c9183057601d270dcc09507a087dd"} Apr 16 13:57:12.821780 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:12.821727 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6srr9" event={"ID":"bbd94c79-236c-4a37-b6ad-384d38491b9e","Type":"ContainerStarted","Data":"90b9d080cc1c2a76acb118dc1b777e2bcc613312160e5060c796c287f354da30"} Apr 16 13:57:12.828191 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:12.828166 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27wm7" event={"ID":"d29c2bfc-de60-442e-a4cb-615b6688e211","Type":"ContainerStarted","Data":"ce54622af3fa106e7e5d84f805ee99d56033b052456554bced0c3465b328ef39"} Apr 16 13:57:12.833122 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:12.833091 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-114.ec2.internal" event={"ID":"031b74a3d41a1b92a3331318a0772b2d","Type":"ContainerStarted","Data":"9e6e62a1a9db073f534c0c5734cbedc6efe6965fbefc8c1a6836fa151f75f9b2"} Apr 16 13:57:12.834529 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:12.834506 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sflxw" event={"ID":"cab34005-67cd-4db7-80d0-1d5b72338051","Type":"ContainerStarted","Data":"844a27c63895e1aafe7ec32d18dc33c2f1c01c80e6d1f6d9b3deda1e507b5979"} Apr 16 13:57:12.836343 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:12.836320 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-brqdj" event={"ID":"48194990-0de3-45a9-acc5-6fcc4c2760d0","Type":"ContainerStarted","Data":"7561c9c68530a9f6f85ac9c37643bcbbb807aac466200239a10f4f923e67fd3e"} Apr 16 13:57:12.988131 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:12.988101 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc18aa469_e518_4e2b_9d80_85e7e1d8ed94.slice/crio-e074ef6219f86916c3f55dbaac9344992ae3d0a4ec0b3e8512bfba0d1fa92833 WatchSource:0}: Error finding container e074ef6219f86916c3f55dbaac9344992ae3d0a4ec0b3e8512bfba0d1fa92833: Status 404 returned error can't find the container with id e074ef6219f86916c3f55dbaac9344992ae3d0a4ec0b3e8512bfba0d1fa92833 Apr 16 13:57:13.021131 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:13.021107 2574 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Apr 16 13:57:13.274373 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:13.274299 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e-metrics-certs\") pod \"network-metrics-daemon-5fh28\" (UID: \"2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e\") " pod="openshift-multus/network-metrics-daemon-5fh28" Apr 16 13:57:13.274526 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:13.274456 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Apr 16 13:57:13.274526 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:13.274516 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e-metrics-certs podName:2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e nodeName:}" failed. No retries permitted until 2026-04-16 13:57:15.27449795 +0000 UTC m=+5.104232801 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e-metrics-certs") pod "network-metrics-daemon-5fh28" (UID: "2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e") : object "openshift-multus"/"metrics-daemon-secret" not registered Apr 16 13:57:13.376083 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:13.376053 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lhh6k\" (UniqueName: \"kubernetes.io/projected/df2f75cb-df70-4c82-b805-f93bcc7e8030-kube-api-access-lhh6k\") pod \"network-check-target-h4r6r\" (UID: \"df2f75cb-df70-4c82-b805-f93bcc7e8030\") " pod="openshift-network-diagnostics/network-check-target-h4r6r" Apr 16 13:57:13.376268 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:13.376234 2574 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Apr 16 13:57:13.376268 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:13.376258 2574 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Apr 16 13:57:13.376383 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:13.376271 2574 projected.go:194] Error preparing data for projected volume kube-api-access-lhh6k for pod openshift-network-diagnostics/network-check-target-h4r6r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 16 13:57:13.376383 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:13.376326 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df2f75cb-df70-4c82-b805-f93bcc7e8030-kube-api-access-lhh6k podName:df2f75cb-df70-4c82-b805-f93bcc7e8030 nodeName:}" failed. No retries permitted until 2026-04-16 13:57:15.376308621 +0000 UTC m=+5.206043483 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-lhh6k" (UniqueName: "kubernetes.io/projected/df2f75cb-df70-4c82-b805-f93bcc7e8030-kube-api-access-lhh6k") pod "network-check-target-h4r6r" (UID: "df2f75cb-df70-4c82-b805-f93bcc7e8030") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 16 13:57:13.695131 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:13.694890 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-04-15 13:52:11 +0000 UTC" deadline="2028-01-03 20:17:01.301922604 +0000 UTC" Apr 16 13:57:13.695131 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:13.694923 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15054h19m47.607003413s" Apr 16 13:57:13.840095 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:13.840057 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" event={"ID":"c18aa469-e518-4e2b-9d80-85e7e1d8ed94","Type":"ContainerStarted","Data":"e074ef6219f86916c3f55dbaac9344992ae3d0a4ec0b3e8512bfba0d1fa92833"} Apr 16 13:57:14.805003 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:14.804594 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5fh28" Apr 16 13:57:14.805003 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:14.804734 2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5fh28" podUID="2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e" Apr 16 13:57:14.805003 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:14.804885 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4r6r" Apr 16 13:57:14.805003 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:14.804981 2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4r6r" podUID="df2f75cb-df70-4c82-b805-f93bcc7e8030" Apr 16 13:57:15.290649 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:15.290558 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e-metrics-certs\") pod \"network-metrics-daemon-5fh28\" (UID: \"2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e\") " pod="openshift-multus/network-metrics-daemon-5fh28" Apr 16 13:57:15.290823 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:15.290704 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Apr 16 13:57:15.290823 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:15.290797 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e-metrics-certs podName:2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e nodeName:}" failed. No retries permitted until 2026-04-16 13:57:19.290776006 +0000 UTC m=+9.120510855 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e-metrics-certs") pod "network-metrics-daemon-5fh28" (UID: "2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e") : object "openshift-multus"/"metrics-daemon-secret" not registered Apr 16 13:57:15.391038 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:15.391002 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lhh6k\" (UniqueName: \"kubernetes.io/projected/df2f75cb-df70-4c82-b805-f93bcc7e8030-kube-api-access-lhh6k\") pod \"network-check-target-h4r6r\" (UID: \"df2f75cb-df70-4c82-b805-f93bcc7e8030\") " pod="openshift-network-diagnostics/network-check-target-h4r6r" Apr 16 13:57:15.391200 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:15.391162 2574 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Apr 16 13:57:15.391200 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:15.391189 2574 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Apr 16 13:57:15.391284 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:15.391204 2574 projected.go:194] Error preparing data for projected volume kube-api-access-lhh6k for pod openshift-network-diagnostics/network-check-target-h4r6r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 16 13:57:15.391284 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:15.391273 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df2f75cb-df70-4c82-b805-f93bcc7e8030-kube-api-access-lhh6k podName:df2f75cb-df70-4c82-b805-f93bcc7e8030 nodeName:}" failed. No retries permitted until 2026-04-16 13:57:19.391250705 +0000 UTC m=+9.220985574 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-lhh6k" (UniqueName: "kubernetes.io/projected/df2f75cb-df70-4c82-b805-f93bcc7e8030-kube-api-access-lhh6k") pod "network-check-target-h4r6r" (UID: "df2f75cb-df70-4c82-b805-f93bcc7e8030") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 16 13:57:16.805087 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:16.805050 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5fh28" Apr 16 13:57:16.805526 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:16.805188 2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5fh28" podUID="2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e" Apr 16 13:57:16.805707 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:16.805685 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4r6r" Apr 16 13:57:16.805831 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:16.805809 2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4r6r" podUID="df2f75cb-df70-4c82-b805-f93bcc7e8030" Apr 16 13:57:17.085427 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:17.085339 2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-lk2d2"] Apr 16 13:57:17.090414 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:17.090382 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lk2d2" Apr 16 13:57:17.090563 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:17.090462 2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lk2d2" podUID="2ac404a2-5e6f-4a47-bc40-f8f182cd968e" Apr 16 13:57:17.201191 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:17.201161 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2ac404a2-5e6f-4a47-bc40-f8f182cd968e-dbus\") pod \"global-pull-secret-syncer-lk2d2\" (UID: \"2ac404a2-5e6f-4a47-bc40-f8f182cd968e\") " pod="kube-system/global-pull-secret-syncer-lk2d2" Apr 16 13:57:17.201191 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:17.201193 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2ac404a2-5e6f-4a47-bc40-f8f182cd968e-original-pull-secret\") pod \"global-pull-secret-syncer-lk2d2\" (UID: \"2ac404a2-5e6f-4a47-bc40-f8f182cd968e\") " pod="kube-system/global-pull-secret-syncer-lk2d2" Apr 16 13:57:17.201391 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:17.201281 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2ac404a2-5e6f-4a47-bc40-f8f182cd968e-kubelet-config\") pod \"global-pull-secret-syncer-lk2d2\" (UID: \"2ac404a2-5e6f-4a47-bc40-f8f182cd968e\") " pod="kube-system/global-pull-secret-syncer-lk2d2" Apr 16 13:57:17.302570 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:17.302532 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2ac404a2-5e6f-4a47-bc40-f8f182cd968e-kubelet-config\") pod \"global-pull-secret-syncer-lk2d2\" (UID: \"2ac404a2-5e6f-4a47-bc40-f8f182cd968e\") " pod="kube-system/global-pull-secret-syncer-lk2d2" Apr 16 13:57:17.302734 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:17.302581 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2ac404a2-5e6f-4a47-bc40-f8f182cd968e-dbus\") pod \"global-pull-secret-syncer-lk2d2\" (UID: \"2ac404a2-5e6f-4a47-bc40-f8f182cd968e\") " pod="kube-system/global-pull-secret-syncer-lk2d2" Apr 16 13:57:17.302734 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:17.302607 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2ac404a2-5e6f-4a47-bc40-f8f182cd968e-original-pull-secret\") pod \"global-pull-secret-syncer-lk2d2\" (UID: \"2ac404a2-5e6f-4a47-bc40-f8f182cd968e\") " pod="kube-system/global-pull-secret-syncer-lk2d2" Apr 16 13:57:17.302734 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:17.302680 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2ac404a2-5e6f-4a47-bc40-f8f182cd968e-kubelet-config\") pod \"global-pull-secret-syncer-lk2d2\" (UID: \"2ac404a2-5e6f-4a47-bc40-f8f182cd968e\") " pod="kube-system/global-pull-secret-syncer-lk2d2" Apr 16 13:57:17.302946 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:17.302763 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Apr 16 13:57:17.302946 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:17.302786 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2ac404a2-5e6f-4a47-bc40-f8f182cd968e-dbus\") pod \"global-pull-secret-syncer-lk2d2\" (UID: \"2ac404a2-5e6f-4a47-bc40-f8f182cd968e\") " pod="kube-system/global-pull-secret-syncer-lk2d2" Apr 16 13:57:17.302946 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:17.302822 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ac404a2-5e6f-4a47-bc40-f8f182cd968e-original-pull-secret podName:2ac404a2-5e6f-4a47-bc40-f8f182cd968e nodeName:}" failed. No retries permitted until 2026-04-16 13:57:17.802802314 +0000 UTC m=+7.632537162 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2ac404a2-5e6f-4a47-bc40-f8f182cd968e-original-pull-secret") pod "global-pull-secret-syncer-lk2d2" (UID: "2ac404a2-5e6f-4a47-bc40-f8f182cd968e") : object "kube-system"/"original-pull-secret" not registered Apr 16 13:57:17.805927 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:17.805889 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2ac404a2-5e6f-4a47-bc40-f8f182cd968e-original-pull-secret\") pod \"global-pull-secret-syncer-lk2d2\" (UID: \"2ac404a2-5e6f-4a47-bc40-f8f182cd968e\") " pod="kube-system/global-pull-secret-syncer-lk2d2" Apr 16 13:57:17.806379 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:17.806045 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Apr 16 13:57:17.806379 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:17.806122 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ac404a2-5e6f-4a47-bc40-f8f182cd968e-original-pull-secret podName:2ac404a2-5e6f-4a47-bc40-f8f182cd968e nodeName:}" failed. No retries permitted until 2026-04-16 13:57:18.806102382 +0000 UTC m=+8.635837241 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2ac404a2-5e6f-4a47-bc40-f8f182cd968e-original-pull-secret") pod "global-pull-secret-syncer-lk2d2" (UID: "2ac404a2-5e6f-4a47-bc40-f8f182cd968e") : object "kube-system"/"original-pull-secret" not registered Apr 16 13:57:18.804847 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:18.804812 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lk2d2" Apr 16 13:57:18.805023 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:18.804938 2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lk2d2" podUID="2ac404a2-5e6f-4a47-bc40-f8f182cd968e" Apr 16 13:57:18.805023 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:18.804966 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5fh28" Apr 16 13:57:18.805141 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:18.805075 2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5fh28" podUID="2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e" Apr 16 13:57:18.805231 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:18.805216 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4r6r" Apr 16 13:57:18.805332 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:18.805298 2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4r6r" podUID="df2f75cb-df70-4c82-b805-f93bcc7e8030" Apr 16 13:57:18.814013 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:18.813986 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2ac404a2-5e6f-4a47-bc40-f8f182cd968e-original-pull-secret\") pod \"global-pull-secret-syncer-lk2d2\" (UID: \"2ac404a2-5e6f-4a47-bc40-f8f182cd968e\") " pod="kube-system/global-pull-secret-syncer-lk2d2" Apr 16 13:57:18.814330 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:18.814124 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Apr 16 13:57:18.814330 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:18.814176 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ac404a2-5e6f-4a47-bc40-f8f182cd968e-original-pull-secret podName:2ac404a2-5e6f-4a47-bc40-f8f182cd968e nodeName:}" failed. No retries permitted until 2026-04-16 13:57:20.814159513 +0000 UTC m=+10.643894366 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2ac404a2-5e6f-4a47-bc40-f8f182cd968e-original-pull-secret") pod "global-pull-secret-syncer-lk2d2" (UID: "2ac404a2-5e6f-4a47-bc40-f8f182cd968e") : object "kube-system"/"original-pull-secret" not registered Apr 16 13:57:18.905311 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:18.905202 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0db6df7e5746d57207c9078241175f3c.slice/crio-4c129604ba00913fdcfc871d781cfafed258e6a7060c9bdf9238f6c19591bfbb WatchSource:0}: Error finding container 4c129604ba00913fdcfc871d781cfafed258e6a7060c9bdf9238f6c19591bfbb: Status 404 returned error can't find the container with id 4c129604ba00913fdcfc871d781cfafed258e6a7060c9bdf9238f6c19591bfbb Apr 16 13:57:18.914708 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:18.914681 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b83df07_d346_4caa_b358_294994a3d89b.slice/crio-2d89cc61eaae4daf54981b5c43ad76d7278a0a38b4ed1614aa27431cb95d97a0 WatchSource:0}: Error finding container 2d89cc61eaae4daf54981b5c43ad76d7278a0a38b4ed1614aa27431cb95d97a0: Status 404 returned error can't find the container with id 2d89cc61eaae4daf54981b5c43ad76d7278a0a38b4ed1614aa27431cb95d97a0 Apr 16 13:57:19.317131 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:19.316878 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e-metrics-certs\") pod \"network-metrics-daemon-5fh28\" (UID: \"2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e\") " pod="openshift-multus/network-metrics-daemon-5fh28" Apr 16 13:57:19.317314 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:19.317222 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Apr 16 13:57:19.317314 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:19.317302 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e-metrics-certs podName:2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e nodeName:}" failed. No retries permitted until 2026-04-16 13:57:27.317281345 +0000 UTC m=+17.147016197 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e-metrics-certs") pod "network-metrics-daemon-5fh28" (UID: "2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e") : object "openshift-multus"/"metrics-daemon-secret" not registered Apr 16 13:57:19.418348 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:19.418312 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lhh6k\" (UniqueName: \"kubernetes.io/projected/df2f75cb-df70-4c82-b805-f93bcc7e8030-kube-api-access-lhh6k\") pod \"network-check-target-h4r6r\" (UID: \"df2f75cb-df70-4c82-b805-f93bcc7e8030\") " pod="openshift-network-diagnostics/network-check-target-h4r6r" Apr 16 13:57:19.418533 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:19.418490 2574 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Apr 16 13:57:19.418533 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:19.418513 2574 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Apr 16 13:57:19.418533 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:19.418525 2574 projected.go:194] Error preparing data for projected volume kube-api-access-lhh6k for pod openshift-network-diagnostics/network-check-target-h4r6r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 16 13:57:19.418688 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:19.418587 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df2f75cb-df70-4c82-b805-f93bcc7e8030-kube-api-access-lhh6k podName:df2f75cb-df70-4c82-b805-f93bcc7e8030 nodeName:}" failed. No retries permitted until 2026-04-16 13:57:27.41856798 +0000 UTC m=+17.248302846 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-lhh6k" (UniqueName: "kubernetes.io/projected/df2f75cb-df70-4c82-b805-f93bcc7e8030-kube-api-access-lhh6k") pod "network-check-target-h4r6r" (UID: "df2f75cb-df70-4c82-b805-f93bcc7e8030") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 16 13:57:19.851195 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:19.851117 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sflxw" event={"ID":"cab34005-67cd-4db7-80d0-1d5b72338051","Type":"ContainerStarted","Data":"df813829161a1daddc20a04e890f8b374036f1fa1aa33d7447d53075897264c7"} Apr 16 13:57:19.852300 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:19.852273 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8js6c" event={"ID":"5b83df07-d346-4caa-b358-294994a3d89b","Type":"ContainerStarted","Data":"2d89cc61eaae4daf54981b5c43ad76d7278a0a38b4ed1614aa27431cb95d97a0"} Apr 16 13:57:19.853404 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:19.853375 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-114.ec2.internal" event={"ID":"0db6df7e5746d57207c9078241175f3c","Type":"ContainerStarted","Data":"4c129604ba00913fdcfc871d781cfafed258e6a7060c9bdf9238f6c19591bfbb"} Apr 16 13:57:19.855141 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:19.855111 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-x4qcz" event={"ID":"c590b6a7-f5a5-476f-a0bd-fb8e8853b8cb","Type":"ContainerStarted","Data":"b7d6ecd359538bc8de8bd8324e55b047796696750659c8e9729df81153b7999e"} Apr 16 13:57:19.857266 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:19.857243 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-brqdj" event={"ID":"48194990-0de3-45a9-acc5-6fcc4c2760d0","Type":"ContainerStarted","Data":"5b0150452f60983bf83f3a87d9d372e2f15a3094203cf0b5f7fd0d5538326623"} Apr 16 13:57:19.858648 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:19.858604 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6srr9" event={"ID":"bbd94c79-236c-4a37-b6ad-384d38491b9e","Type":"ContainerStarted","Data":"e5c89a14912bf356d2999bfae8a402c0720cb0773618477563268f5260fae567"} Apr 16 13:57:19.860112 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:19.860085 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27wm7" event={"ID":"d29c2bfc-de60-442e-a4cb-615b6688e211","Type":"ContainerStarted","Data":"e11dc66e858a81a271acd3d37cfc3aa40b7e4fe4f8f85038fb3dd4543a74648a"} Apr 16 13:57:19.861406 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:19.861383 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-114.ec2.internal" event={"ID":"031b74a3d41a1b92a3331318a0772b2d","Type":"ContainerStarted","Data":"ec9165a998f24122ac23b2847ed2fb32c00569fbfb8140af6b3d20e3c7b75519"} Apr 16 13:57:19.863534 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:19.863483 2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-sflxw" podStartSLOduration=3.018922118 podStartE2EDuration="9.863471034s" podCreationTimestamp="2026-04-16 13:57:10 +0000 UTC" firstStartedPulling="2026-04-16 13:57:12.01292614 +0000 UTC m=+1.842660992" lastFinishedPulling="2026-04-16 13:57:18.857475045 +0000 UTC m=+8.687209908" observedRunningTime="2026-04-16 13:57:19.863044024 +0000 UTC m=+9.692778895" watchObservedRunningTime="2026-04-16 13:57:19.863471034 +0000 UTC m=+9.693205904" Apr 16 13:57:19.874848 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:19.874277 2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-114.ec2.internal" podStartSLOduration=9.87426219 podStartE2EDuration="9.87426219s" podCreationTimestamp="2026-04-16 13:57:10 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 13:57:19.873677318 +0000 UTC m=+9.703412190" watchObservedRunningTime="2026-04-16 13:57:19.87426219 +0000 UTC m=+9.703997061" Apr 16 13:57:19.884943 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:19.884902 2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-x4qcz" podStartSLOduration=3.091207082 podStartE2EDuration="9.884887381s" podCreationTimestamp="2026-04-16 13:57:10 +0000 UTC" firstStartedPulling="2026-04-16 13:57:12.063785246 +0000 UTC m=+1.893520093" lastFinishedPulling="2026-04-16 13:57:18.857465526 +0000 UTC m=+8.687200392" observedRunningTime="2026-04-16 13:57:19.884487494 +0000 UTC m=+9.714222378" watchObservedRunningTime="2026-04-16 13:57:19.884887381 +0000 UTC m=+9.714622253" Apr 16 13:57:19.898289 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:19.898198 2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-brqdj" podStartSLOduration=3.256706786 podStartE2EDuration="9.898184297s" podCreationTimestamp="2026-04-16 13:57:10 +0000 UTC" firstStartedPulling="2026-04-16 13:57:12.21641385 +0000 UTC m=+2.046148697" lastFinishedPulling="2026-04-16 13:57:18.857891352 +0000 UTC m=+8.687626208" observedRunningTime="2026-04-16 13:57:19.897401369 +0000 UTC m=+9.727136241" watchObservedRunningTime="2026-04-16 13:57:19.898184297 +0000 UTC m=+9.727919169" Apr 16 13:57:20.804927 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:20.804895 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5fh28" Apr 16 13:57:20.805923 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:20.805853 2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5fh28" podUID="2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e" Apr 16 13:57:20.806184 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:20.806164 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4r6r" Apr 16 13:57:20.806284 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:20.806262 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lk2d2" Apr 16 13:57:20.806361 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:20.806280 2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4r6r" podUID="df2f75cb-df70-4c82-b805-f93bcc7e8030" Apr 16 13:57:20.806412 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:20.806350 2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lk2d2" podUID="2ac404a2-5e6f-4a47-bc40-f8f182cd968e" Apr 16 13:57:20.829316 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:20.829286 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2ac404a2-5e6f-4a47-bc40-f8f182cd968e-original-pull-secret\") pod \"global-pull-secret-syncer-lk2d2\" (UID: \"2ac404a2-5e6f-4a47-bc40-f8f182cd968e\") " pod="kube-system/global-pull-secret-syncer-lk2d2" Apr 16 13:57:20.829467 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:20.829446 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Apr 16 13:57:20.829527 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:20.829519 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ac404a2-5e6f-4a47-bc40-f8f182cd968e-original-pull-secret podName:2ac404a2-5e6f-4a47-bc40-f8f182cd968e nodeName:}" failed. No retries permitted until 2026-04-16 13:57:24.829497132 +0000 UTC m=+14.659231993 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2ac404a2-5e6f-4a47-bc40-f8f182cd968e-original-pull-secret") pod "global-pull-secret-syncer-lk2d2" (UID: "2ac404a2-5e6f-4a47-bc40-f8f182cd968e") : object "kube-system"/"original-pull-secret" not registered Apr 16 13:57:20.865259 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:20.865231 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-v2ntl" event={"ID":"7d2abdf5-4927-45c4-9386-af7dc0ed8d90","Type":"ContainerStarted","Data":"010a4ca574b32cac0c3a278d22a8e96e0e0d9d9a17ad9d7e3f5d06f53968f077"} Apr 16 13:57:20.867672 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:20.867633 2574 generic.go:358] "Generic (PLEG): container finished" podID="ce951cb8-753a-4902-be96-172949512289" containerID="c9f53619888647924516a18d77a78e54996df26354ea1a52ad1867ca825f274a" exitCode=0 Apr 16 13:57:20.867814 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:20.867733 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rnvkv" event={"ID":"ce951cb8-753a-4902-be96-172949512289","Type":"ContainerDied","Data":"c9f53619888647924516a18d77a78e54996df26354ea1a52ad1867ca825f274a"} Apr 16 13:57:20.877235 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:20.877179 2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-6srr9" podStartSLOduration=3.98722669 podStartE2EDuration="10.877166839s" podCreationTimestamp="2026-04-16 13:57:10 +0000 UTC" firstStartedPulling="2026-04-16 13:57:12.022115023 +0000 UTC m=+1.851849871" lastFinishedPulling="2026-04-16 13:57:18.912055158 +0000 UTC m=+8.741790020" observedRunningTime="2026-04-16 13:57:19.916592196 +0000 UTC m=+9.746327067" watchObservedRunningTime="2026-04-16 13:57:20.877166839 +0000 UTC m=+10.706901708" Apr 16 13:57:20.896977 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:20.896930 2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-v2ntl" podStartSLOduration=4.083415032 podStartE2EDuration="10.896916622s" podCreationTimestamp="2026-04-16 13:57:10 +0000 UTC" firstStartedPulling="2026-04-16 13:57:12.044319412 +0000 UTC m=+1.874054264" lastFinishedPulling="2026-04-16 13:57:18.857821002 +0000 UTC m=+8.687555854" observedRunningTime="2026-04-16 13:57:20.877529894 +0000 UTC m=+10.707264762" watchObservedRunningTime="2026-04-16 13:57:20.896916622 +0000 UTC m=+10.726651494" Apr 16 13:57:21.872217 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:21.871988 2574 generic.go:358] "Generic (PLEG): container finished" podID="0db6df7e5746d57207c9078241175f3c" containerID="064c0b5897918d2b0c949e92b7a64ed83480afac461d6755b728b8ab337974e7" exitCode=0 Apr 16 13:57:21.872655 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:21.872063 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-114.ec2.internal" event={"ID":"0db6df7e5746d57207c9078241175f3c","Type":"ContainerDied","Data":"064c0b5897918d2b0c949e92b7a64ed83480afac461d6755b728b8ab337974e7"} Apr 16 13:57:22.068957 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:22.068920 2574 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Apr 16 13:57:22.707864 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:22.707731 2574 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-04-16T13:57:22.068937095Z","UUID":"eb039737-6104-4317-bd18-51622ad7c257","Handler":null,"Name":"","Endpoint":""} Apr 16 13:57:22.709873 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:22.709845 2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Apr 16 13:57:22.710011 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:22.709883 2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Apr 16 13:57:22.805563 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:22.805052 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4r6r" Apr 16 13:57:22.805563 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:22.805169 2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4r6r" podUID="df2f75cb-df70-4c82-b805-f93bcc7e8030" Apr 16 13:57:22.805563 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:22.805262 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lk2d2" Apr 16 13:57:22.805563 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:22.805363 2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lk2d2" podUID="2ac404a2-5e6f-4a47-bc40-f8f182cd968e" Apr 16 13:57:22.805563 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:22.805430 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5fh28" Apr 16 13:57:22.805563 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:22.805522 2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5fh28" podUID="2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e" Apr 16 13:57:22.876182 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:22.876139 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27wm7" event={"ID":"d29c2bfc-de60-442e-a4cb-615b6688e211","Type":"ContainerStarted","Data":"eac16e22e7286ac7097ecdb2909d3b23f3c1caf379a95da61cef143bfd8c16ce"} Apr 16 13:57:22.878029 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:22.878002 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-114.ec2.internal" event={"ID":"0db6df7e5746d57207c9078241175f3c","Type":"ContainerStarted","Data":"6c22455ec0bb0b0dc2ce859e1fdfe57e57560e39b2bc9bec350c1e94782dccc2"} Apr 16 13:57:23.413685 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:23.413459 2574 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-sflxw" Apr 16 13:57:23.414410 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:23.414224 2574 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-sflxw" Apr 16 13:57:23.427649 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:23.426987 2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-114.ec2.internal" podStartSLOduration=13.42697222 podStartE2EDuration="13.42697222s" podCreationTimestamp="2026-04-16 13:57:10 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 13:57:22.890013539 +0000 UTC m=+12.719748410" watchObservedRunningTime="2026-04-16 13:57:23.42697222 +0000 UTC m=+13.256707091" Apr 16 13:57:24.805541 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:24.805503 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5fh28" Apr 16 13:57:24.806220 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:24.805503 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4r6r" Apr 16 13:57:24.806220 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:24.805637 2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5fh28" podUID="2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e" Apr 16 13:57:24.806220 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:24.805503 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lk2d2" Apr 16 13:57:24.806220 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:24.805708 2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4r6r" podUID="df2f75cb-df70-4c82-b805-f93bcc7e8030" Apr 16 13:57:24.806220 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:24.805783 2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lk2d2" podUID="2ac404a2-5e6f-4a47-bc40-f8f182cd968e" Apr 16 13:57:24.862479 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:24.862438 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2ac404a2-5e6f-4a47-bc40-f8f182cd968e-original-pull-secret\") pod \"global-pull-secret-syncer-lk2d2\" (UID: \"2ac404a2-5e6f-4a47-bc40-f8f182cd968e\") " pod="kube-system/global-pull-secret-syncer-lk2d2" Apr 16 13:57:24.862628 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:24.862594 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Apr 16 13:57:24.862780 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:24.862756 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ac404a2-5e6f-4a47-bc40-f8f182cd968e-original-pull-secret podName:2ac404a2-5e6f-4a47-bc40-f8f182cd968e nodeName:}" failed. No retries permitted until 2026-04-16 13:57:32.862672338 +0000 UTC m=+22.692407220 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2ac404a2-5e6f-4a47-bc40-f8f182cd968e-original-pull-secret") pod "global-pull-secret-syncer-lk2d2" (UID: "2ac404a2-5e6f-4a47-bc40-f8f182cd968e") : object "kube-system"/"original-pull-secret" not registered Apr 16 13:57:26.805181 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:26.805149 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5fh28" Apr 16 13:57:26.805785 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:26.805152 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4r6r" Apr 16 13:57:26.805785 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:26.805154 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lk2d2" Apr 16 13:57:26.805785 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:26.805312 2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5fh28" podUID="2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e" Apr 16 13:57:26.805785 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:26.805414 2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4r6r" podUID="df2f75cb-df70-4c82-b805-f93bcc7e8030" Apr 16 13:57:26.805785 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:26.805506 2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lk2d2" podUID="2ac404a2-5e6f-4a47-bc40-f8f182cd968e" Apr 16 13:57:27.383009 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:27.382976 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e-metrics-certs\") pod \"network-metrics-daemon-5fh28\" (UID: \"2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e\") " pod="openshift-multus/network-metrics-daemon-5fh28" Apr 16 13:57:27.383154 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:27.383089 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Apr 16 13:57:27.383154 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:27.383154 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e-metrics-certs podName:2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e nodeName:}" failed. No retries permitted until 2026-04-16 13:57:43.383136439 +0000 UTC m=+33.212871290 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e-metrics-certs") pod "network-metrics-daemon-5fh28" (UID: "2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e") : object "openshift-multus"/"metrics-daemon-secret" not registered Apr 16 13:57:27.483967 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:27.483923 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lhh6k\" (UniqueName: \"kubernetes.io/projected/df2f75cb-df70-4c82-b805-f93bcc7e8030-kube-api-access-lhh6k\") pod \"network-check-target-h4r6r\" (UID: \"df2f75cb-df70-4c82-b805-f93bcc7e8030\") " pod="openshift-network-diagnostics/network-check-target-h4r6r" Apr 16 13:57:27.484144 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:27.484110 2574 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Apr 16 13:57:27.484144 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:27.484139 2574 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Apr 16 13:57:27.484254 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:27.484151 2574 projected.go:194] Error preparing data for projected volume kube-api-access-lhh6k for pod openshift-network-diagnostics/network-check-target-h4r6r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 16 13:57:27.484254 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:27.484209 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df2f75cb-df70-4c82-b805-f93bcc7e8030-kube-api-access-lhh6k podName:df2f75cb-df70-4c82-b805-f93bcc7e8030 nodeName:}" failed. No retries permitted until 2026-04-16 13:57:43.484189265 +0000 UTC m=+33.313924169 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-lhh6k" (UniqueName: "kubernetes.io/projected/df2f75cb-df70-4c82-b805-f93bcc7e8030-kube-api-access-lhh6k") pod "network-check-target-h4r6r" (UID: "df2f75cb-df70-4c82-b805-f93bcc7e8030") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 16 13:57:28.805105 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:28.805070 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5fh28" Apr 16 13:57:28.805530 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:28.805203 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4r6r" Apr 16 13:57:28.805530 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:28.805210 2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5fh28" podUID="2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e" Apr 16 13:57:28.805530 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:28.805323 2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4r6r" podUID="df2f75cb-df70-4c82-b805-f93bcc7e8030" Apr 16 13:57:28.805530 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:28.805361 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lk2d2" Apr 16 13:57:28.805530 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:28.805441 2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lk2d2" podUID="2ac404a2-5e6f-4a47-bc40-f8f182cd968e" Apr 16 13:57:30.806363 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:30.806140 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5fh28" Apr 16 13:57:30.806813 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:30.806210 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4r6r" Apr 16 13:57:30.806813 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:30.806442 2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5fh28" podUID="2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e" Apr 16 13:57:30.806813 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:30.806229 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lk2d2" Apr 16 13:57:30.806813 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:30.806531 2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4r6r" podUID="df2f75cb-df70-4c82-b805-f93bcc7e8030" Apr 16 13:57:30.806813 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:30.806611 2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lk2d2" podUID="2ac404a2-5e6f-4a47-bc40-f8f182cd968e" Apr 16 13:57:32.804671 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:32.804640 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5fh28" Apr 16 13:57:32.805028 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:32.804674 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4r6r" Apr 16 13:57:32.805028 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:32.804675 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lk2d2" Apr 16 13:57:32.805028 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:32.804771 2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5fh28" podUID="2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e" Apr 16 13:57:32.805028 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:32.804839 2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4r6r" podUID="df2f75cb-df70-4c82-b805-f93bcc7e8030" Apr 16 13:57:32.805028 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:32.804935 2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lk2d2" podUID="2ac404a2-5e6f-4a47-bc40-f8f182cd968e" Apr 16 13:57:32.832884 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:32.832859 2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-sflxw" Apr 16 13:57:32.832993 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:32.832983 2574 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Apr 16 13:57:32.833698 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:32.833676 2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-sflxw" Apr 16 13:57:32.924519 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:32.924483 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2ac404a2-5e6f-4a47-bc40-f8f182cd968e-original-pull-secret\") pod \"global-pull-secret-syncer-lk2d2\" (UID: \"2ac404a2-5e6f-4a47-bc40-f8f182cd968e\") " pod="kube-system/global-pull-secret-syncer-lk2d2" Apr 16 13:57:32.924732 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:32.924619 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Apr 16 13:57:32.924732 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:32.924688 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ac404a2-5e6f-4a47-bc40-f8f182cd968e-original-pull-secret podName:2ac404a2-5e6f-4a47-bc40-f8f182cd968e nodeName:}" failed. No retries permitted until 2026-04-16 13:57:48.924669036 +0000 UTC m=+38.754403890 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2ac404a2-5e6f-4a47-bc40-f8f182cd968e-original-pull-secret") pod "global-pull-secret-syncer-lk2d2" (UID: "2ac404a2-5e6f-4a47-bc40-f8f182cd968e") : object "kube-system"/"original-pull-secret" not registered Apr 16 13:57:33.897711 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:33.897672 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8js6c" event={"ID":"5b83df07-d346-4caa-b358-294994a3d89b","Type":"ContainerStarted","Data":"c0feaeb1b090e298f7d4c303e428e9cdc013a8c5d938fe25195d66257e95d160"} Apr 16 13:57:33.899307 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:33.899287 2574 generic.go:358] "Generic (PLEG): container finished" podID="ce951cb8-753a-4902-be96-172949512289" containerID="51063e9cbe1bc3d419c60a12707c8617cc21fcbf30b6b76410501df239814517" exitCode=0 Apr 16 13:57:33.899428 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:33.899341 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rnvkv" event={"ID":"ce951cb8-753a-4902-be96-172949512289","Type":"ContainerDied","Data":"51063e9cbe1bc3d419c60a12707c8617cc21fcbf30b6b76410501df239814517"} Apr 16 13:57:33.901870 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:33.901848 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" event={"ID":"c18aa469-e518-4e2b-9d80-85e7e1d8ed94","Type":"ContainerStarted","Data":"79b2a214d875aa059216425c8de63c26e5c6475ec8db6731a6f04219cda7c490"} Apr 16 13:57:33.901949 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:33.901877 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" event={"ID":"c18aa469-e518-4e2b-9d80-85e7e1d8ed94","Type":"ContainerStarted","Data":"bb974f797acccc4e6d59019be957e6cb0494cb3b3812b9dfd95176652fcf5f5a"} Apr 16 13:57:33.901949 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:33.901890 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" event={"ID":"c18aa469-e518-4e2b-9d80-85e7e1d8ed94","Type":"ContainerStarted","Data":"91550f9f042dc1eb5e17c158791a4e6e70278df705107a518a68a12872f88cc6"} Apr 16 13:57:33.901949 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:33.901898 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" event={"ID":"c18aa469-e518-4e2b-9d80-85e7e1d8ed94","Type":"ContainerStarted","Data":"160006de7df2a69d2cfaf12f5ac8b8eaf783aab1a3fd53bbbddffeb6ca30dc2c"} Apr 16 13:57:33.901949 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:33.901906 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" event={"ID":"c18aa469-e518-4e2b-9d80-85e7e1d8ed94","Type":"ContainerStarted","Data":"18fe21b940a1cb1a9e162abb1831f9ac424906435db817e892061806f1a43019"} Apr 16 13:57:33.901949 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:33.901914 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" event={"ID":"c18aa469-e518-4e2b-9d80-85e7e1d8ed94","Type":"ContainerStarted","Data":"6b7f1d61d78a144c6c29d937a73c78b1e63cecbda22b7983fe73138706d8c2ae"} Apr 16 13:57:33.903495 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:33.903471 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27wm7" event={"ID":"d29c2bfc-de60-442e-a4cb-615b6688e211","Type":"ContainerStarted","Data":"047fb450d5a24a2a386bf04b9fe7101b7cac026125eb52f5a375f8682535f323"} Apr 16 13:57:33.916222 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:33.915056 2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-8js6c" podStartSLOduration=9.86214006 podStartE2EDuration="23.915040803s" podCreationTimestamp="2026-04-16 13:57:10 +0000 UTC" firstStartedPulling="2026-04-16 13:57:18.916395734 +0000 UTC m=+8.746130595" lastFinishedPulling="2026-04-16 13:57:32.96929649 +0000 UTC m=+22.799031338" observedRunningTime="2026-04-16 13:57:33.914341328 +0000 UTC m=+23.744076203" watchObservedRunningTime="2026-04-16 13:57:33.915040803 +0000 UTC m=+23.744775673" Apr 16 13:57:33.950563 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:33.950526 2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27wm7" podStartSLOduration=2.95480194 podStartE2EDuration="23.95051312s" podCreationTimestamp="2026-04-16 13:57:10 +0000 UTC" firstStartedPulling="2026-04-16 13:57:12.012925184 +0000 UTC m=+1.842660039" lastFinishedPulling="2026-04-16 13:57:33.008636355 +0000 UTC m=+22.838371219" observedRunningTime="2026-04-16 13:57:33.929886356 +0000 UTC m=+23.759621235" watchObservedRunningTime="2026-04-16 13:57:33.95051312 +0000 UTC m=+23.780247987" Apr 16 13:57:34.805472 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:34.805443 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lk2d2" Apr 16 13:57:34.805635 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:34.805555 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4r6r" Apr 16 13:57:34.805635 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:34.805564 2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lk2d2" podUID="2ac404a2-5e6f-4a47-bc40-f8f182cd968e" Apr 16 13:57:34.805635 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:34.805628 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5fh28" Apr 16 13:57:34.805734 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:34.805645 2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4r6r" podUID="df2f75cb-df70-4c82-b805-f93bcc7e8030" Apr 16 13:57:34.805734 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:34.805697 2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5fh28" podUID="2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e" Apr 16 13:57:35.908106 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:35.908068 2574 generic.go:358] "Generic (PLEG): container finished" podID="ce951cb8-753a-4902-be96-172949512289" containerID="9fbe506a51c1273ef21ebbb20149726cdc91671707d5e25631ffebedba412d20" exitCode=0 Apr 16 13:57:35.908792 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:35.908120 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rnvkv" event={"ID":"ce951cb8-753a-4902-be96-172949512289","Type":"ContainerDied","Data":"9fbe506a51c1273ef21ebbb20149726cdc91671707d5e25631ffebedba412d20"} Apr 16 13:57:35.910923 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:35.910897 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" event={"ID":"c18aa469-e518-4e2b-9d80-85e7e1d8ed94","Type":"ContainerStarted","Data":"19c7a5762def2252db27ce4f801ccad369986e8c7cc79b06274f77c44665ee8d"} Apr 16 13:57:36.805500 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:36.805473 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lk2d2" Apr 16 13:57:36.805589 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:36.805484 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5fh28" Apr 16 13:57:36.805650 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:36.805588 2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lk2d2" podUID="2ac404a2-5e6f-4a47-bc40-f8f182cd968e" Apr 16 13:57:36.805708 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:36.805484 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4r6r" Apr 16 13:57:36.805708 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:36.805654 2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5fh28" podUID="2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e" Apr 16 13:57:36.805830 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:36.805713 2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4r6r" podUID="df2f75cb-df70-4c82-b805-f93bcc7e8030" Apr 16 13:57:36.914604 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:36.914539 2574 generic.go:358] "Generic (PLEG): container finished" podID="ce951cb8-753a-4902-be96-172949512289" containerID="b850dad32a4319ce7f10586b2c72f1711c56eb2a9933a2693ac7ad1860ccc11f" exitCode=0 Apr 16 13:57:36.914911 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:36.914592 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rnvkv" event={"ID":"ce951cb8-753a-4902-be96-172949512289","Type":"ContainerDied","Data":"b850dad32a4319ce7f10586b2c72f1711c56eb2a9933a2693ac7ad1860ccc11f"} Apr 16 13:57:37.920603 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:37.920272 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" event={"ID":"c18aa469-e518-4e2b-9d80-85e7e1d8ed94","Type":"ContainerStarted","Data":"257a40d177cceea86bb8207c14bbd0df167838ca01f750e77d6b4369ac465f91"} Apr 16 13:57:37.920603 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:37.920526 2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" Apr 16 13:57:37.920603 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:37.920546 2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" Apr 16 13:57:37.920603 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:37.920561 2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" Apr 16 13:57:37.937481 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:37.937453 2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" Apr 16 13:57:37.938039 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:37.938021 2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" Apr 16 13:57:37.946195 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:37.946155 2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" podStartSLOduration=8.142661618 podStartE2EDuration="27.946141147s" podCreationTimestamp="2026-04-16 13:57:10 +0000 UTC" firstStartedPulling="2026-04-16 13:57:12.99006818 +0000 UTC m=+2.819803040" lastFinishedPulling="2026-04-16 13:57:32.79354772 +0000 UTC m=+22.623282569" observedRunningTime="2026-04-16 13:57:37.94547776 +0000 UTC m=+27.775212630" watchObservedRunningTime="2026-04-16 13:57:37.946141147 +0000 UTC m=+27.775876017" Apr 16 13:57:38.805146 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:38.805103 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5fh28" Apr 16 13:57:38.805322 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:38.805103 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lk2d2" Apr 16 13:57:38.805322 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:38.805231 2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5fh28" podUID="2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e" Apr 16 13:57:38.805322 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:38.805122 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4r6r" Apr 16 13:57:38.805322 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:38.805267 2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lk2d2" podUID="2ac404a2-5e6f-4a47-bc40-f8f182cd968e" Apr 16 13:57:38.805484 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:38.805383 2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4r6r" podUID="df2f75cb-df70-4c82-b805-f93bcc7e8030" Apr 16 13:57:39.504175 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:39.503980 2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-h4r6r"] Apr 16 13:57:39.504175 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:39.504091 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4r6r" Apr 16 13:57:39.504693 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:39.504191 2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4r6r" podUID="df2f75cb-df70-4c82-b805-f93bcc7e8030" Apr 16 13:57:39.505125 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:39.504916 2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lk2d2"] Apr 16 13:57:39.505125 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:39.504995 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lk2d2" Apr 16 13:57:39.505125 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:39.505084 2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lk2d2" podUID="2ac404a2-5e6f-4a47-bc40-f8f182cd968e" Apr 16 13:57:39.505698 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:39.505601 2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5fh28"] Apr 16 13:57:39.505698 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:39.505680 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5fh28" Apr 16 13:57:39.506070 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:39.505805 2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5fh28" podUID="2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e" Apr 16 13:57:40.805823 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:40.805798 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lk2d2" Apr 16 13:57:40.806190 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:40.805877 2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lk2d2" podUID="2ac404a2-5e6f-4a47-bc40-f8f182cd968e" Apr 16 13:57:40.806190 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:40.805958 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4r6r" Apr 16 13:57:40.806190 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:40.806045 2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4r6r" podUID="df2f75cb-df70-4c82-b805-f93bcc7e8030" Apr 16 13:57:41.804873 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:41.804844 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5fh28" Apr 16 13:57:41.805020 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:41.804971 2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5fh28" podUID="2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e" Apr 16 13:57:42.804887 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:42.804852 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lk2d2" Apr 16 13:57:42.805316 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:42.804850 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4r6r" Apr 16 13:57:42.805316 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:42.804973 2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lk2d2" podUID="2ac404a2-5e6f-4a47-bc40-f8f182cd968e" Apr 16 13:57:42.805316 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:42.805037 2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4r6r" podUID="df2f75cb-df70-4c82-b805-f93bcc7e8030" Apr 16 13:57:43.402303 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:43.402170 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e-metrics-certs\") pod \"network-metrics-daemon-5fh28\" (UID: \"2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e\") " pod="openshift-multus/network-metrics-daemon-5fh28" Apr 16 13:57:43.402398 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:43.402301 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Apr 16 13:57:43.402398 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:43.402362 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e-metrics-certs podName:2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e nodeName:}" failed. No retries permitted until 2026-04-16 13:58:15.40234318 +0000 UTC m=+65.232078037 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e-metrics-certs") pod "network-metrics-daemon-5fh28" (UID: "2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e") : object "openshift-multus"/"metrics-daemon-secret" not registered Apr 16 13:57:43.503542 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:43.503515 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lhh6k\" (UniqueName: \"kubernetes.io/projected/df2f75cb-df70-4c82-b805-f93bcc7e8030-kube-api-access-lhh6k\") pod \"network-check-target-h4r6r\" (UID: \"df2f75cb-df70-4c82-b805-f93bcc7e8030\") " pod="openshift-network-diagnostics/network-check-target-h4r6r" Apr 16 13:57:43.503705 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:43.503686 2574 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Apr 16 13:57:43.503764 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:43.503709 2574 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Apr 16 13:57:43.503764 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:43.503719 2574 projected.go:194] Error preparing data for projected volume kube-api-access-lhh6k for pod openshift-network-diagnostics/network-check-target-h4r6r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 16 13:57:43.503832 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:43.503779 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df2f75cb-df70-4c82-b805-f93bcc7e8030-kube-api-access-lhh6k podName:df2f75cb-df70-4c82-b805-f93bcc7e8030 nodeName:}" failed. No retries permitted until 2026-04-16 13:58:15.50376091 +0000 UTC m=+65.333495773 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-lhh6k" (UniqueName: "kubernetes.io/projected/df2f75cb-df70-4c82-b805-f93bcc7e8030-kube-api-access-lhh6k") pod "network-check-target-h4r6r" (UID: "df2f75cb-df70-4c82-b805-f93bcc7e8030") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 16 13:57:43.804476 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:43.804452 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5fh28" Apr 16 13:57:43.804629 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:43.804557 2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5fh28" podUID="2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e" Apr 16 13:57:43.933152 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:43.933079 2574 generic.go:358] "Generic (PLEG): container finished" podID="ce951cb8-753a-4902-be96-172949512289" containerID="c9abc1ec4fa6e9eb86bad1af93e3db9312cd69b764fcaa641926a47ff3d57e22" exitCode=0 Apr 16 13:57:43.933575 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:43.933149 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rnvkv" event={"ID":"ce951cb8-753a-4902-be96-172949512289","Type":"ContainerDied","Data":"c9abc1ec4fa6e9eb86bad1af93e3db9312cd69b764fcaa641926a47ff3d57e22"} Apr 16 13:57:44.804726 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:44.804696 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4r6r" Apr 16 13:57:44.804935 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:44.804808 2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4r6r" podUID="df2f75cb-df70-4c82-b805-f93bcc7e8030" Apr 16 13:57:44.804935 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:44.804850 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lk2d2" Apr 16 13:57:44.804935 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:44.804900 2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lk2d2" podUID="2ac404a2-5e6f-4a47-bc40-f8f182cd968e" Apr 16 13:57:44.937427 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:44.937398 2574 generic.go:358] "Generic (PLEG): container finished" podID="ce951cb8-753a-4902-be96-172949512289" containerID="ebd5c149ce43d3ad5b0847a52364e66176148bc95255fa97e97fce1607c530ec" exitCode=0 Apr 16 13:57:44.937906 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:44.937447 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rnvkv" event={"ID":"ce951cb8-753a-4902-be96-172949512289","Type":"ContainerDied","Data":"ebd5c149ce43d3ad5b0847a52364e66176148bc95255fa97e97fce1607c530ec"} Apr 16 13:57:45.461258 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:45.461223 2574 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-114.ec2.internal" event="NodeReady" Apr 16 13:57:45.461416 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:45.461338 2574 kubelet_node_status.go:550] "Fast updating node status as it just became ready" Apr 16 13:57:45.504816 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:45.504777 2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-lnfdm"] Apr 16 13:57:45.527958 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:45.527683 2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xjpgz"] Apr 16 13:57:45.543146 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:45.543119 2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xjpgz"] Apr 16 13:57:45.543146 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:45.543142 2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lnfdm"] Apr 16 13:57:45.543308 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:45.543227 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xjpgz" Apr 16 13:57:45.543308 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:45.543233 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lnfdm" Apr 16 13:57:45.545577 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:45.545558 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Apr 16 13:57:45.545710 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:45.545682 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Apr 16 13:57:45.545847 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:45.545814 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Apr 16 13:57:45.545913 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:45.545846 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Apr 16 13:57:45.546004 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:45.545956 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-mv45m\"" Apr 16 13:57:45.546223 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:45.546209 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Apr 16 13:57:45.546299 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:45.546274 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ftzrr\"" Apr 16 13:57:45.630462 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:45.630409 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ee1c9f7-5758-4734-a7b3-ea621cc6df95-cert\") pod \"ingress-canary-xjpgz\" (UID: \"7ee1c9f7-5758-4734-a7b3-ea621cc6df95\") " pod="openshift-ingress-canary/ingress-canary-xjpgz" Apr 16 13:57:45.630462 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:45.630437 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eded8e93-5cbc-435c-aab3-a909812ba63b-metrics-tls\") pod \"dns-default-lnfdm\" (UID: \"eded8e93-5cbc-435c-aab3-a909812ba63b\") " pod="openshift-dns/dns-default-lnfdm" Apr 16 13:57:45.630610 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:45.630473 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eded8e93-5cbc-435c-aab3-a909812ba63b-tmp-dir\") pod \"dns-default-lnfdm\" (UID: \"eded8e93-5cbc-435c-aab3-a909812ba63b\") " pod="openshift-dns/dns-default-lnfdm" Apr 16 13:57:45.630610 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:45.630527 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxqrw\" (UniqueName: \"kubernetes.io/projected/7ee1c9f7-5758-4734-a7b3-ea621cc6df95-kube-api-access-sxqrw\") pod \"ingress-canary-xjpgz\" (UID: \"7ee1c9f7-5758-4734-a7b3-ea621cc6df95\") " pod="openshift-ingress-canary/ingress-canary-xjpgz" Apr 16 13:57:45.630688 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:45.630635 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr7p8\" (UniqueName: \"kubernetes.io/projected/eded8e93-5cbc-435c-aab3-a909812ba63b-kube-api-access-pr7p8\") pod \"dns-default-lnfdm\" (UID: \"eded8e93-5cbc-435c-aab3-a909812ba63b\") " pod="openshift-dns/dns-default-lnfdm" Apr 16 13:57:45.630688 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:45.630664 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/eded8e93-5cbc-435c-aab3-a909812ba63b-config-volume\") pod \"dns-default-lnfdm\" (UID: \"eded8e93-5cbc-435c-aab3-a909812ba63b\") " pod="openshift-dns/dns-default-lnfdm" Apr 16 13:57:45.730983 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:45.730959 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pr7p8\" (UniqueName: \"kubernetes.io/projected/eded8e93-5cbc-435c-aab3-a909812ba63b-kube-api-access-pr7p8\") pod \"dns-default-lnfdm\" (UID: \"eded8e93-5cbc-435c-aab3-a909812ba63b\") " pod="openshift-dns/dns-default-lnfdm" Apr 16 13:57:45.731111 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:45.730988 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/eded8e93-5cbc-435c-aab3-a909812ba63b-config-volume\") pod \"dns-default-lnfdm\" (UID: \"eded8e93-5cbc-435c-aab3-a909812ba63b\") " pod="openshift-dns/dns-default-lnfdm" Apr 16 13:57:45.731111 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:45.731005 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ee1c9f7-5758-4734-a7b3-ea621cc6df95-cert\") pod \"ingress-canary-xjpgz\" (UID: \"7ee1c9f7-5758-4734-a7b3-ea621cc6df95\") " pod="openshift-ingress-canary/ingress-canary-xjpgz" Apr 16 13:57:45.731111 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:45.731022 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eded8e93-5cbc-435c-aab3-a909812ba63b-metrics-tls\") pod \"dns-default-lnfdm\" (UID: \"eded8e93-5cbc-435c-aab3-a909812ba63b\") " pod="openshift-dns/dns-default-lnfdm" Apr 16 13:57:45.731111 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:45.731094 2574 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found Apr 16 13:57:45.731313 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:45.731126 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eded8e93-5cbc-435c-aab3-a909812ba63b-tmp-dir\") pod \"dns-default-lnfdm\" (UID: \"eded8e93-5cbc-435c-aab3-a909812ba63b\") " pod="openshift-dns/dns-default-lnfdm" Apr 16 13:57:45.731313 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:45.731141 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ee1c9f7-5758-4734-a7b3-ea621cc6df95-cert podName:7ee1c9f7-5758-4734-a7b3-ea621cc6df95 nodeName:}" failed. No retries permitted until 2026-04-16 13:57:46.231127487 +0000 UTC m=+36.060862334 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/7ee1c9f7-5758-4734-a7b3-ea621cc6df95-cert") pod "ingress-canary-xjpgz" (UID: "7ee1c9f7-5758-4734-a7b3-ea621cc6df95") : secret "canary-serving-cert" not found Apr 16 13:57:45.731313 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:45.731173 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-sxqrw\" (UniqueName: \"kubernetes.io/projected/7ee1c9f7-5758-4734-a7b3-ea621cc6df95-kube-api-access-sxqrw\") pod \"ingress-canary-xjpgz\" (UID: \"7ee1c9f7-5758-4734-a7b3-ea621cc6df95\") " pod="openshift-ingress-canary/ingress-canary-xjpgz" Apr 16 13:57:45.731477 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:45.731356 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Apr 16 13:57:45.731477 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:45.731401 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eded8e93-5cbc-435c-aab3-a909812ba63b-tmp-dir\") pod \"dns-default-lnfdm\" (UID: \"eded8e93-5cbc-435c-aab3-a909812ba63b\") " pod="openshift-dns/dns-default-lnfdm" Apr 16 13:57:45.731477 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:45.731411 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eded8e93-5cbc-435c-aab3-a909812ba63b-metrics-tls podName:eded8e93-5cbc-435c-aab3-a909812ba63b nodeName:}" failed. No retries permitted until 2026-04-16 13:57:46.231394501 +0000 UTC m=+36.061129354 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/eded8e93-5cbc-435c-aab3-a909812ba63b-metrics-tls") pod "dns-default-lnfdm" (UID: "eded8e93-5cbc-435c-aab3-a909812ba63b") : secret "dns-default-metrics-tls" not found Apr 16 13:57:45.731659 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:45.731637 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/eded8e93-5cbc-435c-aab3-a909812ba63b-config-volume\") pod \"dns-default-lnfdm\" (UID: \"eded8e93-5cbc-435c-aab3-a909812ba63b\") " pod="openshift-dns/dns-default-lnfdm" Apr 16 13:57:45.746526 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:45.746506 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr7p8\" (UniqueName: \"kubernetes.io/projected/eded8e93-5cbc-435c-aab3-a909812ba63b-kube-api-access-pr7p8\") pod \"dns-default-lnfdm\" (UID: \"eded8e93-5cbc-435c-aab3-a909812ba63b\") " pod="openshift-dns/dns-default-lnfdm" Apr 16 13:57:45.746661 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:45.746570 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxqrw\" (UniqueName: \"kubernetes.io/projected/7ee1c9f7-5758-4734-a7b3-ea621cc6df95-kube-api-access-sxqrw\") pod \"ingress-canary-xjpgz\" (UID: \"7ee1c9f7-5758-4734-a7b3-ea621cc6df95\") " pod="openshift-ingress-canary/ingress-canary-xjpgz" Apr 16 13:57:45.805518 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:45.805490 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5fh28" Apr 16 13:57:45.808011 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:45.807992 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jz268\"" Apr 16 13:57:45.808112 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:45.807993 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Apr 16 13:57:45.941856 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:45.941779 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rnvkv" event={"ID":"ce951cb8-753a-4902-be96-172949512289","Type":"ContainerStarted","Data":"bb24de3eb560545acfdf414f2bf00deb3d0cd25abc2a5d9bc148dd6a2ae2bc24"} Apr 16 13:57:45.967114 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:45.967064 2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-rnvkv" podStartSLOduration=4.622424453 podStartE2EDuration="35.967047993s" podCreationTimestamp="2026-04-16 13:57:10 +0000 UTC" firstStartedPulling="2026-04-16 13:57:12.026830252 +0000 UTC m=+1.856565100" lastFinishedPulling="2026-04-16 13:57:43.371453782 +0000 UTC m=+33.201188640" observedRunningTime="2026-04-16 13:57:45.963170593 +0000 UTC m=+35.792905454" watchObservedRunningTime="2026-04-16 13:57:45.967047993 +0000 UTC m=+35.796782862" Apr 16 13:57:46.234433 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:46.234356 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ee1c9f7-5758-4734-a7b3-ea621cc6df95-cert\") pod \"ingress-canary-xjpgz\" (UID: \"7ee1c9f7-5758-4734-a7b3-ea621cc6df95\") " pod="openshift-ingress-canary/ingress-canary-xjpgz" Apr 16 13:57:46.234433 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:46.234395 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eded8e93-5cbc-435c-aab3-a909812ba63b-metrics-tls\") pod \"dns-default-lnfdm\" (UID: \"eded8e93-5cbc-435c-aab3-a909812ba63b\") " pod="openshift-dns/dns-default-lnfdm" Apr 16 13:57:46.234671 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:46.234490 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Apr 16 13:57:46.234671 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:46.234494 2574 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found Apr 16 13:57:46.234671 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:46.234543 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eded8e93-5cbc-435c-aab3-a909812ba63b-metrics-tls podName:eded8e93-5cbc-435c-aab3-a909812ba63b nodeName:}" failed. No retries permitted until 2026-04-16 13:57:47.234528687 +0000 UTC m=+37.064263536 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/eded8e93-5cbc-435c-aab3-a909812ba63b-metrics-tls") pod "dns-default-lnfdm" (UID: "eded8e93-5cbc-435c-aab3-a909812ba63b") : secret "dns-default-metrics-tls" not found Apr 16 13:57:46.234671 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:46.234556 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ee1c9f7-5758-4734-a7b3-ea621cc6df95-cert podName:7ee1c9f7-5758-4734-a7b3-ea621cc6df95 nodeName:}" failed. No retries permitted until 2026-04-16 13:57:47.234550373 +0000 UTC m=+37.064285220 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/7ee1c9f7-5758-4734-a7b3-ea621cc6df95-cert") pod "ingress-canary-xjpgz" (UID: "7ee1c9f7-5758-4734-a7b3-ea621cc6df95") : secret "canary-serving-cert" not found Apr 16 13:57:46.805139 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:46.805106 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lk2d2" Apr 16 13:57:46.805302 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:46.805106 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4r6r" Apr 16 13:57:46.807712 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:46.807692 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Apr 16 13:57:46.808471 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:46.808450 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7n6t4\"" Apr 16 13:57:46.808573 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:46.808472 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Apr 16 13:57:46.808573 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:46.808525 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Apr 16 13:57:47.242404 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:47.242371 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ee1c9f7-5758-4734-a7b3-ea621cc6df95-cert\") pod \"ingress-canary-xjpgz\" (UID: \"7ee1c9f7-5758-4734-a7b3-ea621cc6df95\") " pod="openshift-ingress-canary/ingress-canary-xjpgz" Apr 16 13:57:47.242404 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:47.242408 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eded8e93-5cbc-435c-aab3-a909812ba63b-metrics-tls\") pod \"dns-default-lnfdm\" (UID: \"eded8e93-5cbc-435c-aab3-a909812ba63b\") " pod="openshift-dns/dns-default-lnfdm" Apr 16 13:57:47.242819 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:47.242501 2574 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found Apr 16 13:57:47.242819 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:47.242510 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Apr 16 13:57:47.242819 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:47.242566 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eded8e93-5cbc-435c-aab3-a909812ba63b-metrics-tls podName:eded8e93-5cbc-435c-aab3-a909812ba63b nodeName:}" failed. No retries permitted until 2026-04-16 13:57:49.242552284 +0000 UTC m=+39.072287132 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/eded8e93-5cbc-435c-aab3-a909812ba63b-metrics-tls") pod "dns-default-lnfdm" (UID: "eded8e93-5cbc-435c-aab3-a909812ba63b") : secret "dns-default-metrics-tls" not found Apr 16 13:57:47.242819 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:47.242579 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ee1c9f7-5758-4734-a7b3-ea621cc6df95-cert podName:7ee1c9f7-5758-4734-a7b3-ea621cc6df95 nodeName:}" failed. No retries permitted until 2026-04-16 13:57:49.242573191 +0000 UTC m=+39.072308038 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/7ee1c9f7-5758-4734-a7b3-ea621cc6df95-cert") pod "ingress-canary-xjpgz" (UID: "7ee1c9f7-5758-4734-a7b3-ea621cc6df95") : secret "canary-serving-cert" not found Apr 16 13:57:48.952671 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:48.952643 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2ac404a2-5e6f-4a47-bc40-f8f182cd968e-original-pull-secret\") pod \"global-pull-secret-syncer-lk2d2\" (UID: \"2ac404a2-5e6f-4a47-bc40-f8f182cd968e\") " pod="kube-system/global-pull-secret-syncer-lk2d2" Apr 16 13:57:48.954929 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:48.954913 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2ac404a2-5e6f-4a47-bc40-f8f182cd968e-original-pull-secret\") pod \"global-pull-secret-syncer-lk2d2\" (UID: \"2ac404a2-5e6f-4a47-bc40-f8f182cd968e\") " pod="kube-system/global-pull-secret-syncer-lk2d2" Apr 16 13:57:49.214954 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:49.214900 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lk2d2" Apr 16 13:57:49.254886 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:49.254854 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ee1c9f7-5758-4734-a7b3-ea621cc6df95-cert\") pod \"ingress-canary-xjpgz\" (UID: \"7ee1c9f7-5758-4734-a7b3-ea621cc6df95\") " pod="openshift-ingress-canary/ingress-canary-xjpgz" Apr 16 13:57:49.255005 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:49.254901 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eded8e93-5cbc-435c-aab3-a909812ba63b-metrics-tls\") pod \"dns-default-lnfdm\" (UID: \"eded8e93-5cbc-435c-aab3-a909812ba63b\") " pod="openshift-dns/dns-default-lnfdm" Apr 16 13:57:49.255050 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:49.255001 2574 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found Apr 16 13:57:49.255082 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:49.255069 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ee1c9f7-5758-4734-a7b3-ea621cc6df95-cert podName:7ee1c9f7-5758-4734-a7b3-ea621cc6df95 nodeName:}" failed. No retries permitted until 2026-04-16 13:57:53.255055083 +0000 UTC m=+43.084789931 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/7ee1c9f7-5758-4734-a7b3-ea621cc6df95-cert") pod "ingress-canary-xjpgz" (UID: "7ee1c9f7-5758-4734-a7b3-ea621cc6df95") : secret "canary-serving-cert" not found Apr 16 13:57:49.255133 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:49.255005 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Apr 16 13:57:49.255167 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:49.255158 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eded8e93-5cbc-435c-aab3-a909812ba63b-metrics-tls podName:eded8e93-5cbc-435c-aab3-a909812ba63b nodeName:}" failed. No retries permitted until 2026-04-16 13:57:53.255141611 +0000 UTC m=+43.084876460 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/eded8e93-5cbc-435c-aab3-a909812ba63b-metrics-tls") pod "dns-default-lnfdm" (UID: "eded8e93-5cbc-435c-aab3-a909812ba63b") : secret "dns-default-metrics-tls" not found Apr 16 13:57:49.389935 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:49.389729 2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lk2d2"] Apr 16 13:57:49.401970 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:57:49.401925 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ac404a2_5e6f_4a47_bc40_f8f182cd968e.slice/crio-6485f3233b4e9bf98b1a6014999bbb8cecbd0ff4fa8e23983ad5f77e243c59cc WatchSource:0}: Error finding container 6485f3233b4e9bf98b1a6014999bbb8cecbd0ff4fa8e23983ad5f77e243c59cc: Status 404 returned error can't find the container with id 6485f3233b4e9bf98b1a6014999bbb8cecbd0ff4fa8e23983ad5f77e243c59cc Apr 16 13:57:49.950938 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:49.950893 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lk2d2" event={"ID":"2ac404a2-5e6f-4a47-bc40-f8f182cd968e","Type":"ContainerStarted","Data":"6485f3233b4e9bf98b1a6014999bbb8cecbd0ff4fa8e23983ad5f77e243c59cc"} Apr 16 13:57:53.283903 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:53.283810 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ee1c9f7-5758-4734-a7b3-ea621cc6df95-cert\") pod \"ingress-canary-xjpgz\" (UID: \"7ee1c9f7-5758-4734-a7b3-ea621cc6df95\") " pod="openshift-ingress-canary/ingress-canary-xjpgz" Apr 16 13:57:53.283903 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:53.283859 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eded8e93-5cbc-435c-aab3-a909812ba63b-metrics-tls\") pod \"dns-default-lnfdm\" (UID: \"eded8e93-5cbc-435c-aab3-a909812ba63b\") " pod="openshift-dns/dns-default-lnfdm" Apr 16 13:57:53.284299 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:53.283965 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Apr 16 13:57:53.284299 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:53.284019 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eded8e93-5cbc-435c-aab3-a909812ba63b-metrics-tls podName:eded8e93-5cbc-435c-aab3-a909812ba63b nodeName:}" failed. No retries permitted until 2026-04-16 13:58:01.284000396 +0000 UTC m=+51.113735259 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/eded8e93-5cbc-435c-aab3-a909812ba63b-metrics-tls") pod "dns-default-lnfdm" (UID: "eded8e93-5cbc-435c-aab3-a909812ba63b") : secret "dns-default-metrics-tls" not found Apr 16 13:57:53.284299 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:53.283965 2574 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found Apr 16 13:57:53.284299 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:57:53.284088 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ee1c9f7-5758-4734-a7b3-ea621cc6df95-cert podName:7ee1c9f7-5758-4734-a7b3-ea621cc6df95 nodeName:}" failed. No retries permitted until 2026-04-16 13:58:01.284075359 +0000 UTC m=+51.113810207 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/7ee1c9f7-5758-4734-a7b3-ea621cc6df95-cert") pod "ingress-canary-xjpgz" (UID: "7ee1c9f7-5758-4734-a7b3-ea621cc6df95") : secret "canary-serving-cert" not found Apr 16 13:57:53.959506 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:53.959467 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lk2d2" event={"ID":"2ac404a2-5e6f-4a47-bc40-f8f182cd968e","Type":"ContainerStarted","Data":"6acc9a65b4bebc6c346eb944a625f6ebdb445889ddd136e43ceefe3c84d35643"} Apr 16 13:57:53.973970 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:57:53.973929 2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-lk2d2" podStartSLOduration=33.403914899 podStartE2EDuration="36.973915991s" podCreationTimestamp="2026-04-16 13:57:17 +0000 UTC" firstStartedPulling="2026-04-16 13:57:49.40498696 +0000 UTC m=+39.234721809" lastFinishedPulling="2026-04-16 13:57:52.974988042 +0000 UTC m=+42.804722901" observedRunningTime="2026-04-16 13:57:53.973353656 +0000 UTC m=+43.803088562" watchObservedRunningTime="2026-04-16 13:57:53.973915991 +0000 UTC m=+43.803650860" Apr 16 13:58:01.341922 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:58:01.341887 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ee1c9f7-5758-4734-a7b3-ea621cc6df95-cert\") pod \"ingress-canary-xjpgz\" (UID: \"7ee1c9f7-5758-4734-a7b3-ea621cc6df95\") " pod="openshift-ingress-canary/ingress-canary-xjpgz" Apr 16 13:58:01.341922 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:58:01.341924 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eded8e93-5cbc-435c-aab3-a909812ba63b-metrics-tls\") pod \"dns-default-lnfdm\" (UID: \"eded8e93-5cbc-435c-aab3-a909812ba63b\") " pod="openshift-dns/dns-default-lnfdm" Apr 16 13:58:01.342310 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:58:01.342030 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Apr 16 13:58:01.342310 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:58:01.342033 2574 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found Apr 16 13:58:01.342310 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:58:01.342078 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eded8e93-5cbc-435c-aab3-a909812ba63b-metrics-tls podName:eded8e93-5cbc-435c-aab3-a909812ba63b nodeName:}" failed. No retries permitted until 2026-04-16 13:58:17.342065542 +0000 UTC m=+67.171800389 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/eded8e93-5cbc-435c-aab3-a909812ba63b-metrics-tls") pod "dns-default-lnfdm" (UID: "eded8e93-5cbc-435c-aab3-a909812ba63b") : secret "dns-default-metrics-tls" not found Apr 16 13:58:01.342310 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:58:01.342092 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ee1c9f7-5758-4734-a7b3-ea621cc6df95-cert podName:7ee1c9f7-5758-4734-a7b3-ea621cc6df95 nodeName:}" failed. No retries permitted until 2026-04-16 13:58:17.342085206 +0000 UTC m=+67.171820054 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/7ee1c9f7-5758-4734-a7b3-ea621cc6df95-cert") pod "ingress-canary-xjpgz" (UID: "7ee1c9f7-5758-4734-a7b3-ea621cc6df95") : secret "canary-serving-cert" not found Apr 16 13:58:09.943785 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:58:09.943731 2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-99w5m" Apr 16 13:58:15.428126 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:58:15.428082 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e-metrics-certs\") pod \"network-metrics-daemon-5fh28\" (UID: \"2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e\") " pod="openshift-multus/network-metrics-daemon-5fh28" Apr 16 13:58:15.430573 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:58:15.430552 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Apr 16 13:58:15.438448 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:58:15.438426 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Apr 16 13:58:15.438505 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:58:15.438495 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e-metrics-certs podName:2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e nodeName:}" failed. No retries permitted until 2026-04-16 13:59:19.438480428 +0000 UTC m=+129.268215275 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e-metrics-certs") pod "network-metrics-daemon-5fh28" (UID: "2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e") : secret "metrics-daemon-secret" not found Apr 16 13:58:15.529379 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:58:15.529350 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lhh6k\" (UniqueName: \"kubernetes.io/projected/df2f75cb-df70-4c82-b805-f93bcc7e8030-kube-api-access-lhh6k\") pod \"network-check-target-h4r6r\" (UID: \"df2f75cb-df70-4c82-b805-f93bcc7e8030\") " pod="openshift-network-diagnostics/network-check-target-h4r6r" Apr 16 13:58:15.531779 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:58:15.531762 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Apr 16 13:58:15.541510 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:58:15.541493 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Apr 16 13:58:15.553472 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:58:15.553449 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhh6k\" (UniqueName: \"kubernetes.io/projected/df2f75cb-df70-4c82-b805-f93bcc7e8030-kube-api-access-lhh6k\") pod \"network-check-target-h4r6r\" (UID: \"df2f75cb-df70-4c82-b805-f93bcc7e8030\") " pod="openshift-network-diagnostics/network-check-target-h4r6r" Apr 16 13:58:15.621809 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:58:15.621788 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7n6t4\"" Apr 16 13:58:15.629648 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:58:15.629632 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4r6r" Apr 16 13:58:15.735860 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:58:15.735834 2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-h4r6r"] Apr 16 13:58:15.739374 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:58:15.739347 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf2f75cb_df70_4c82_b805_f93bcc7e8030.slice/crio-15003cb53de42cbcac4ebb9f907f062070bed9c3ff0dbb543901d8005d0c85ff WatchSource:0}: Error finding container 15003cb53de42cbcac4ebb9f907f062070bed9c3ff0dbb543901d8005d0c85ff: Status 404 returned error can't find the container with id 15003cb53de42cbcac4ebb9f907f062070bed9c3ff0dbb543901d8005d0c85ff Apr 16 13:58:15.996099 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:58:15.996017 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-h4r6r" event={"ID":"df2f75cb-df70-4c82-b805-f93bcc7e8030","Type":"ContainerStarted","Data":"15003cb53de42cbcac4ebb9f907f062070bed9c3ff0dbb543901d8005d0c85ff"} Apr 16 13:58:17.441645 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:58:17.441605 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ee1c9f7-5758-4734-a7b3-ea621cc6df95-cert\") pod \"ingress-canary-xjpgz\" (UID: \"7ee1c9f7-5758-4734-a7b3-ea621cc6df95\") " pod="openshift-ingress-canary/ingress-canary-xjpgz" Apr 16 13:58:17.442068 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:58:17.441654 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eded8e93-5cbc-435c-aab3-a909812ba63b-metrics-tls\") pod \"dns-default-lnfdm\" (UID: \"eded8e93-5cbc-435c-aab3-a909812ba63b\") " pod="openshift-dns/dns-default-lnfdm" Apr 16 13:58:17.442068 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:58:17.441785 2574 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found Apr 16 13:58:17.442068 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:58:17.441799 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Apr 16 13:58:17.442068 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:58:17.441859 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ee1c9f7-5758-4734-a7b3-ea621cc6df95-cert podName:7ee1c9f7-5758-4734-a7b3-ea621cc6df95 nodeName:}" failed. No retries permitted until 2026-04-16 13:58:49.441842967 +0000 UTC m=+99.271577816 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/7ee1c9f7-5758-4734-a7b3-ea621cc6df95-cert") pod "ingress-canary-xjpgz" (UID: "7ee1c9f7-5758-4734-a7b3-ea621cc6df95") : secret "canary-serving-cert" not found Apr 16 13:58:17.442068 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:58:17.441874 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eded8e93-5cbc-435c-aab3-a909812ba63b-metrics-tls podName:eded8e93-5cbc-435c-aab3-a909812ba63b nodeName:}" failed. No retries permitted until 2026-04-16 13:58:49.441867877 +0000 UTC m=+99.271602724 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/eded8e93-5cbc-435c-aab3-a909812ba63b-metrics-tls") pod "dns-default-lnfdm" (UID: "eded8e93-5cbc-435c-aab3-a909812ba63b") : secret "dns-default-metrics-tls" not found Apr 16 13:58:20.005227 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:58:20.005194 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-h4r6r" event={"ID":"df2f75cb-df70-4c82-b805-f93bcc7e8030","Type":"ContainerStarted","Data":"da93646e420e75a84725f5e168c3fdb675669ddd091c816f4205bffe2391ee29"} Apr 16 13:58:20.005654 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:58:20.005291 2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-h4r6r" Apr 16 13:58:20.020894 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:58:20.020853 2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-h4r6r" podStartSLOduration=66.418456465 podStartE2EDuration="1m10.020839327s" podCreationTimestamp="2026-04-16 13:57:10 +0000 UTC" firstStartedPulling="2026-04-16 13:58:15.74111695 +0000 UTC m=+65.570851802" lastFinishedPulling="2026-04-16 13:58:19.343499817 +0000 UTC m=+69.173234664" observedRunningTime="2026-04-16 13:58:20.020018144 +0000 UTC m=+69.849753027" watchObservedRunningTime="2026-04-16 13:58:20.020839327 +0000 UTC m=+69.850574197" Apr 16 13:58:22.340211 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:58:22.340104 2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-88bf968cd-kjf66"] Apr 16 13:58:22.342909 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:58:22.342894 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-88bf968cd-kjf66" Apr 16 13:58:22.346436 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:58:22.345453 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Apr 16 13:58:22.346436 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:58:22.346089 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Apr 16 13:58:22.346639 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:58:22.346552 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Apr 16 13:58:22.346909 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:58:22.346889 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Apr 16 13:58:22.357864 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:58:22.357844 2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-88bf968cd-kjf66"] Apr 16 13:58:22.377110 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:58:22.377086 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2680aac5-ec4b-407c-9893-addf70d8ca2f-tmp\") pod \"klusterlet-addon-workmgr-88bf968cd-kjf66\" (UID: \"2680aac5-ec4b-407c-9893-addf70d8ca2f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-88bf968cd-kjf66" Apr 16 13:58:22.377208 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:58:22.377115 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2680aac5-ec4b-407c-9893-addf70d8ca2f-klusterlet-config\") pod \"klusterlet-addon-workmgr-88bf968cd-kjf66\" (UID: \"2680aac5-ec4b-407c-9893-addf70d8ca2f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-88bf968cd-kjf66" Apr 16 13:58:22.377208 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:58:22.377134 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jql67\" (UniqueName: \"kubernetes.io/projected/2680aac5-ec4b-407c-9893-addf70d8ca2f-kube-api-access-jql67\") pod \"klusterlet-addon-workmgr-88bf968cd-kjf66\" (UID: \"2680aac5-ec4b-407c-9893-addf70d8ca2f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-88bf968cd-kjf66" Apr 16 13:58:22.477822 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:58:22.477786 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2680aac5-ec4b-407c-9893-addf70d8ca2f-tmp\") pod \"klusterlet-addon-workmgr-88bf968cd-kjf66\" (UID: \"2680aac5-ec4b-407c-9893-addf70d8ca2f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-88bf968cd-kjf66" Apr 16 13:58:22.477962 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:58:22.477826 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2680aac5-ec4b-407c-9893-addf70d8ca2f-klusterlet-config\") pod \"klusterlet-addon-workmgr-88bf968cd-kjf66\" (UID: \"2680aac5-ec4b-407c-9893-addf70d8ca2f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-88bf968cd-kjf66" Apr 16 13:58:22.477962 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:58:22.477848 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jql67\" (UniqueName: \"kubernetes.io/projected/2680aac5-ec4b-407c-9893-addf70d8ca2f-kube-api-access-jql67\") pod \"klusterlet-addon-workmgr-88bf968cd-kjf66\" (UID: \"2680aac5-ec4b-407c-9893-addf70d8ca2f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-88bf968cd-kjf66" Apr 16 13:58:22.478191 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:58:22.478169 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2680aac5-ec4b-407c-9893-addf70d8ca2f-tmp\") pod \"klusterlet-addon-workmgr-88bf968cd-kjf66\" (UID: \"2680aac5-ec4b-407c-9893-addf70d8ca2f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-88bf968cd-kjf66" Apr 16 13:58:22.480283 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:58:22.480264 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2680aac5-ec4b-407c-9893-addf70d8ca2f-klusterlet-config\") pod \"klusterlet-addon-workmgr-88bf968cd-kjf66\" (UID: \"2680aac5-ec4b-407c-9893-addf70d8ca2f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-88bf968cd-kjf66" Apr 16 13:58:22.485508 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:58:22.485486 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jql67\" (UniqueName: \"kubernetes.io/projected/2680aac5-ec4b-407c-9893-addf70d8ca2f-kube-api-access-jql67\") pod \"klusterlet-addon-workmgr-88bf968cd-kjf66\" (UID: \"2680aac5-ec4b-407c-9893-addf70d8ca2f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-88bf968cd-kjf66" Apr 16 13:58:22.654138 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:58:22.654048 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-88bf968cd-kjf66" Apr 16 13:58:22.764763 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:58:22.764671 2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-88bf968cd-kjf66"] Apr 16 13:58:22.768596 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:58:22.768571 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2680aac5_ec4b_407c_9893_addf70d8ca2f.slice/crio-6353c0ce2c61d59e918cfab199df6037dce32eebc7940213949d9eaea9276823 WatchSource:0}: Error finding container 6353c0ce2c61d59e918cfab199df6037dce32eebc7940213949d9eaea9276823: Status 404 returned error can't find the container with id 6353c0ce2c61d59e918cfab199df6037dce32eebc7940213949d9eaea9276823 Apr 16 13:58:23.011466 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:58:23.011387 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-88bf968cd-kjf66" event={"ID":"2680aac5-ec4b-407c-9893-addf70d8ca2f","Type":"ContainerStarted","Data":"6353c0ce2c61d59e918cfab199df6037dce32eebc7940213949d9eaea9276823"} Apr 16 13:58:28.023270 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:58:28.023183 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-88bf968cd-kjf66" event={"ID":"2680aac5-ec4b-407c-9893-addf70d8ca2f","Type":"ContainerStarted","Data":"eb8b593fbd8fb72eb12639ba8921c40d04553bccd47f3c4da74378d1b434ad8b"} Apr 16 13:58:28.023689 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:58:28.023373 2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-88bf968cd-kjf66" Apr 16 13:58:28.025040 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:58:28.025021 2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-88bf968cd-kjf66" Apr 16 13:58:28.037993 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:58:28.037951 2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-88bf968cd-kjf66" podStartSLOduration=1.134519596 podStartE2EDuration="6.037940807s" podCreationTimestamp="2026-04-16 13:58:22 +0000 UTC" firstStartedPulling="2026-04-16 13:58:22.770726482 +0000 UTC m=+72.600461333" lastFinishedPulling="2026-04-16 13:58:27.674147692 +0000 UTC m=+77.503882544" observedRunningTime="2026-04-16 13:58:28.037449512 +0000 UTC m=+77.867184602" watchObservedRunningTime="2026-04-16 13:58:28.037940807 +0000 UTC m=+77.867675676" Apr 16 13:58:49.455128 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:58:49.455090 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ee1c9f7-5758-4734-a7b3-ea621cc6df95-cert\") pod \"ingress-canary-xjpgz\" (UID: \"7ee1c9f7-5758-4734-a7b3-ea621cc6df95\") " pod="openshift-ingress-canary/ingress-canary-xjpgz" Apr 16 13:58:49.455128 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:58:49.455132 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eded8e93-5cbc-435c-aab3-a909812ba63b-metrics-tls\") pod \"dns-default-lnfdm\" (UID: \"eded8e93-5cbc-435c-aab3-a909812ba63b\") " pod="openshift-dns/dns-default-lnfdm" Apr 16 13:58:49.455541 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:58:49.455216 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Apr 16 13:58:49.455541 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:58:49.455222 2574 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found Apr 16 13:58:49.455541 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:58:49.455266 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eded8e93-5cbc-435c-aab3-a909812ba63b-metrics-tls podName:eded8e93-5cbc-435c-aab3-a909812ba63b nodeName:}" failed. No retries permitted until 2026-04-16 13:59:53.455252658 +0000 UTC m=+163.284987506 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/eded8e93-5cbc-435c-aab3-a909812ba63b-metrics-tls") pod "dns-default-lnfdm" (UID: "eded8e93-5cbc-435c-aab3-a909812ba63b") : secret "dns-default-metrics-tls" not found Apr 16 13:58:49.455541 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:58:49.455279 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ee1c9f7-5758-4734-a7b3-ea621cc6df95-cert podName:7ee1c9f7-5758-4734-a7b3-ea621cc6df95 nodeName:}" failed. No retries permitted until 2026-04-16 13:59:53.455273885 +0000 UTC m=+163.285008732 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/7ee1c9f7-5758-4734-a7b3-ea621cc6df95-cert") pod "ingress-canary-xjpgz" (UID: "7ee1c9f7-5758-4734-a7b3-ea621cc6df95") : secret "canary-serving-cert" not found Apr 16 13:58:51.009626 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:58:51.009596 2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-h4r6r" Apr 16 13:59:19.447907 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:19.447865 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e-metrics-certs\") pod \"network-metrics-daemon-5fh28\" (UID: \"2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e\") " pod="openshift-multus/network-metrics-daemon-5fh28" Apr 16 13:59:19.448365 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:59:19.447976 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Apr 16 13:59:19.448365 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:59:19.448025 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e-metrics-certs podName:2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e nodeName:}" failed. No retries permitted until 2026-04-16 14:01:21.448011611 +0000 UTC m=+251.277746458 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e-metrics-certs") pod "network-metrics-daemon-5fh28" (UID: "2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e") : secret "metrics-daemon-secret" not found Apr 16 13:59:42.889439 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:42.889406 2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-69cb8777b6-kbr8q"] Apr 16 13:59:42.892268 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:42.892252 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-69cb8777b6-kbr8q" Apr 16 13:59:42.900271 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:42.900252 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Apr 16 13:59:42.900386 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:42.900259 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Apr 16 13:59:42.900941 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:42.900927 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Apr 16 13:59:42.901087 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:42.901066 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Apr 16 13:59:42.901227 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:42.901083 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Apr 16 13:59:42.901315 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:42.901110 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Apr 16 13:59:42.908626 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:42.908609 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-mzmln\"" Apr 16 13:59:42.921774 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:42.921735 2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-69cb8777b6-kbr8q"] Apr 16 13:59:42.991814 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:42.991790 2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-b76cd7dd-grjcr"] Apr 16 13:59:42.994454 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:42.994440 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b76cd7dd-grjcr" Apr 16 13:59:42.994818 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:42.994799 2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-69965bb79d-hzmxv"] Apr 16 13:59:42.997649 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:42.997630 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-69965bb79d-hzmxv" Apr 16 13:59:42.999088 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:42.999072 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-qt6qg\"" Apr 16 13:59:42.999394 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:42.999378 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Apr 16 13:59:42.999847 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:42.999830 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e81d5797-d78f-46cf-b792-fdb668a624fc-metrics-certs\") pod \"router-default-69cb8777b6-kbr8q\" (UID: \"e81d5797-d78f-46cf-b792-fdb668a624fc\") " pod="openshift-ingress/router-default-69cb8777b6-kbr8q" Apr 16 13:59:42.999933 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:42.999871 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e81d5797-d78f-46cf-b792-fdb668a624fc-default-certificate\") pod \"router-default-69cb8777b6-kbr8q\" (UID: \"e81d5797-d78f-46cf-b792-fdb668a624fc\") " pod="openshift-ingress/router-default-69cb8777b6-kbr8q" Apr 16 13:59:42.999933 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:42.999896 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e81d5797-d78f-46cf-b792-fdb668a624fc-stats-auth\") pod \"router-default-69cb8777b6-kbr8q\" (UID: \"e81d5797-d78f-46cf-b792-fdb668a624fc\") " pod="openshift-ingress/router-default-69cb8777b6-kbr8q" Apr 16 13:59:43.000088 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:42.999987 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e81d5797-d78f-46cf-b792-fdb668a624fc-service-ca-bundle\") pod \"router-default-69cb8777b6-kbr8q\" (UID: \"e81d5797-d78f-46cf-b792-fdb668a624fc\") " pod="openshift-ingress/router-default-69cb8777b6-kbr8q" Apr 16 13:59:43.000088 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.000013 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxlf2\" (UniqueName: \"kubernetes.io/projected/e81d5797-d78f-46cf-b792-fdb668a624fc-kube-api-access-lxlf2\") pod \"router-default-69cb8777b6-kbr8q\" (UID: \"e81d5797-d78f-46cf-b792-fdb668a624fc\") " pod="openshift-ingress/router-default-69cb8777b6-kbr8q" Apr 16 13:59:43.001057 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.001043 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Apr 16 13:59:43.003002 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.002985 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Apr 16 13:59:43.003080 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.002984 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Apr 16 13:59:43.003080 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.003030 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Apr 16 13:59:43.004066 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.004048 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Apr 16 13:59:43.008272 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.008253 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Apr 16 13:59:43.008976 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.008957 2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-69965bb79d-hzmxv"] Apr 16 13:59:43.011494 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.011477 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-vkgkr\"" Apr 16 13:59:43.019627 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.019605 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Apr 16 13:59:43.022704 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.022673 2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-b76cd7dd-grjcr"] Apr 16 13:59:43.083235 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.083211 2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6667474d89-tj5v6"] Apr 16 13:59:43.085906 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.085892 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6667474d89-tj5v6" Apr 16 13:59:43.089849 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.089829 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Apr 16 13:59:43.089849 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.089841 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Apr 16 13:59:43.089849 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.089835 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-x6sn6\"" Apr 16 13:59:43.090044 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.089835 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Apr 16 13:59:43.090044 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.089841 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Apr 16 13:59:43.098217 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.098201 2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6667474d89-tj5v6"] Apr 16 13:59:43.100677 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.100660 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdcsf\" (UniqueName: \"kubernetes.io/projected/eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac-kube-api-access-pdcsf\") pod \"image-registry-b76cd7dd-grjcr\" (UID: \"eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac\") " pod="openshift-image-registry/image-registry-b76cd7dd-grjcr" Apr 16 13:59:43.100774 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.100692 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac-registry-certificates\") pod \"image-registry-b76cd7dd-grjcr\" (UID: \"eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac\") " pod="openshift-image-registry/image-registry-b76cd7dd-grjcr" Apr 16 13:59:43.100842 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.100729 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e81d5797-d78f-46cf-b792-fdb668a624fc-service-ca-bundle\") pod \"router-default-69cb8777b6-kbr8q\" (UID: \"e81d5797-d78f-46cf-b792-fdb668a624fc\") " pod="openshift-ingress/router-default-69cb8777b6-kbr8q" Apr 16 13:59:43.100842 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.100823 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lxlf2\" (UniqueName: \"kubernetes.io/projected/e81d5797-d78f-46cf-b792-fdb668a624fc-kube-api-access-lxlf2\") pod \"router-default-69cb8777b6-kbr8q\" (UID: \"e81d5797-d78f-46cf-b792-fdb668a624fc\") " pod="openshift-ingress/router-default-69cb8777b6-kbr8q" Apr 16 13:59:43.100941 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:59:43.100842 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e81d5797-d78f-46cf-b792-fdb668a624fc-service-ca-bundle podName:e81d5797-d78f-46cf-b792-fdb668a624fc nodeName:}" failed. No retries permitted until 2026-04-16 13:59:43.600827576 +0000 UTC m=+153.430562428 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e81d5797-d78f-46cf-b792-fdb668a624fc-service-ca-bundle") pod "router-default-69cb8777b6-kbr8q" (UID: "e81d5797-d78f-46cf-b792-fdb668a624fc") : configmap references non-existent config key: service-ca.crt Apr 16 13:59:43.100941 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.100877 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86tzr\" (UniqueName: \"kubernetes.io/projected/e9f0f164-4788-43c4-890f-52ed0197b594-kube-api-access-86tzr\") pod \"service-ca-operator-69965bb79d-hzmxv\" (UID: \"e9f0f164-4788-43c4-890f-52ed0197b594\") " pod="openshift-service-ca-operator/service-ca-operator-69965bb79d-hzmxv" Apr 16 13:59:43.100941 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.100905 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac-image-registry-private-configuration\") pod \"image-registry-b76cd7dd-grjcr\" (UID: \"eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac\") " pod="openshift-image-registry/image-registry-b76cd7dd-grjcr" Apr 16 13:59:43.101115 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.100945 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac-installation-pull-secrets\") pod \"image-registry-b76cd7dd-grjcr\" (UID: \"eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac\") " pod="openshift-image-registry/image-registry-b76cd7dd-grjcr" Apr 16 13:59:43.101115 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.100986 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac-registry-tls\") pod \"image-registry-b76cd7dd-grjcr\" (UID: \"eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac\") " pod="openshift-image-registry/image-registry-b76cd7dd-grjcr" Apr 16 13:59:43.101115 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.101025 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e81d5797-d78f-46cf-b792-fdb668a624fc-metrics-certs\") pod \"router-default-69cb8777b6-kbr8q\" (UID: \"e81d5797-d78f-46cf-b792-fdb668a624fc\") " pod="openshift-ingress/router-default-69cb8777b6-kbr8q" Apr 16 13:59:43.101115 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.101048 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e9f0f164-4788-43c4-890f-52ed0197b594-config\") pod \"service-ca-operator-69965bb79d-hzmxv\" (UID: \"e9f0f164-4788-43c4-890f-52ed0197b594\") " pod="openshift-service-ca-operator/service-ca-operator-69965bb79d-hzmxv" Apr 16 13:59:43.101115 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.101085 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac-trusted-ca\") pod \"image-registry-b76cd7dd-grjcr\" (UID: \"eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac\") " pod="openshift-image-registry/image-registry-b76cd7dd-grjcr" Apr 16 13:59:43.101115 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:59:43.101115 2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Apr 16 13:59:43.101396 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.101145 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e81d5797-d78f-46cf-b792-fdb668a624fc-default-certificate\") pod \"router-default-69cb8777b6-kbr8q\" (UID: \"e81d5797-d78f-46cf-b792-fdb668a624fc\") " pod="openshift-ingress/router-default-69cb8777b6-kbr8q" Apr 16 13:59:43.101396 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.101211 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e81d5797-d78f-46cf-b792-fdb668a624fc-stats-auth\") pod \"router-default-69cb8777b6-kbr8q\" (UID: \"e81d5797-d78f-46cf-b792-fdb668a624fc\") " pod="openshift-ingress/router-default-69cb8777b6-kbr8q" Apr 16 13:59:43.101396 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:59:43.101230 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e81d5797-d78f-46cf-b792-fdb668a624fc-metrics-certs podName:e81d5797-d78f-46cf-b792-fdb668a624fc nodeName:}" failed. No retries permitted until 2026-04-16 13:59:43.601213024 +0000 UTC m=+153.430947875 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e81d5797-d78f-46cf-b792-fdb668a624fc-metrics-certs") pod "router-default-69cb8777b6-kbr8q" (UID: "e81d5797-d78f-46cf-b792-fdb668a624fc") : secret "router-metrics-certs-default" not found Apr 16 13:59:43.101396 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.101267 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac-ca-trust-extracted\") pod \"image-registry-b76cd7dd-grjcr\" (UID: \"eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac\") " pod="openshift-image-registry/image-registry-b76cd7dd-grjcr" Apr 16 13:59:43.101396 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.101296 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac-bound-sa-token\") pod \"image-registry-b76cd7dd-grjcr\" (UID: \"eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac\") " pod="openshift-image-registry/image-registry-b76cd7dd-grjcr" Apr 16 13:59:43.101396 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.101329 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e9f0f164-4788-43c4-890f-52ed0197b594-serving-cert\") pod \"service-ca-operator-69965bb79d-hzmxv\" (UID: \"e9f0f164-4788-43c4-890f-52ed0197b594\") " pod="openshift-service-ca-operator/service-ca-operator-69965bb79d-hzmxv" Apr 16 13:59:43.103493 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.103472 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e81d5797-d78f-46cf-b792-fdb668a624fc-default-certificate\") pod \"router-default-69cb8777b6-kbr8q\" (UID: \"e81d5797-d78f-46cf-b792-fdb668a624fc\") " pod="openshift-ingress/router-default-69cb8777b6-kbr8q" Apr 16 13:59:43.103571 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.103537 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e81d5797-d78f-46cf-b792-fdb668a624fc-stats-auth\") pod \"router-default-69cb8777b6-kbr8q\" (UID: \"e81d5797-d78f-46cf-b792-fdb668a624fc\") " pod="openshift-ingress/router-default-69cb8777b6-kbr8q" Apr 16 13:59:43.111353 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.111334 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxlf2\" (UniqueName: \"kubernetes.io/projected/e81d5797-d78f-46cf-b792-fdb668a624fc-kube-api-access-lxlf2\") pod \"router-default-69cb8777b6-kbr8q\" (UID: \"e81d5797-d78f-46cf-b792-fdb668a624fc\") " pod="openshift-ingress/router-default-69cb8777b6-kbr8q" Apr 16 13:59:43.202229 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.202165 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e9f0f164-4788-43c4-890f-52ed0197b594-config\") pod \"service-ca-operator-69965bb79d-hzmxv\" (UID: \"e9f0f164-4788-43c4-890f-52ed0197b594\") " pod="openshift-service-ca-operator/service-ca-operator-69965bb79d-hzmxv" Apr 16 13:59:43.202229 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.202199 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac-bound-sa-token\") pod \"image-registry-b76cd7dd-grjcr\" (UID: \"eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac\") " pod="openshift-image-registry/image-registry-b76cd7dd-grjcr" Apr 16 13:59:43.202382 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.202231 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac-ca-trust-extracted\") pod \"image-registry-b76cd7dd-grjcr\" (UID: \"eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac\") " pod="openshift-image-registry/image-registry-b76cd7dd-grjcr" Apr 16 13:59:43.202382 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.202259 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfdl8\" (UniqueName: \"kubernetes.io/projected/5b9fe9ed-1a79-4590-ad6e-74acb8e02aed-kube-api-access-pfdl8\") pod \"cluster-monitoring-operator-6667474d89-tj5v6\" (UID: \"5b9fe9ed-1a79-4590-ad6e-74acb8e02aed\") " pod="openshift-monitoring/cluster-monitoring-operator-6667474d89-tj5v6" Apr 16 13:59:43.202382 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.202287 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac-registry-certificates\") pod \"image-registry-b76cd7dd-grjcr\" (UID: \"eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac\") " pod="openshift-image-registry/image-registry-b76cd7dd-grjcr" Apr 16 13:59:43.202529 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.202423 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-86tzr\" (UniqueName: \"kubernetes.io/projected/e9f0f164-4788-43c4-890f-52ed0197b594-kube-api-access-86tzr\") pod \"service-ca-operator-69965bb79d-hzmxv\" (UID: \"e9f0f164-4788-43c4-890f-52ed0197b594\") " pod="openshift-service-ca-operator/service-ca-operator-69965bb79d-hzmxv" Apr 16 13:59:43.202529 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.202464 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac-image-registry-private-configuration\") pod \"image-registry-b76cd7dd-grjcr\" (UID: \"eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac\") " pod="openshift-image-registry/image-registry-b76cd7dd-grjcr" Apr 16 13:59:43.202529 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.202502 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac-installation-pull-secrets\") pod \"image-registry-b76cd7dd-grjcr\" (UID: \"eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac\") " pod="openshift-image-registry/image-registry-b76cd7dd-grjcr" Apr 16 13:59:43.202692 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.202538 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac-registry-tls\") pod \"image-registry-b76cd7dd-grjcr\" (UID: \"eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac\") " pod="openshift-image-registry/image-registry-b76cd7dd-grjcr" Apr 16 13:59:43.202692 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.202646 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac-ca-trust-extracted\") pod \"image-registry-b76cd7dd-grjcr\" (UID: \"eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac\") " pod="openshift-image-registry/image-registry-b76cd7dd-grjcr" Apr 16 13:59:43.202827 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:59:43.202725 2574 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Apr 16 13:59:43.202827 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:59:43.202759 2574 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b76cd7dd-grjcr: secret "image-registry-tls" not found Apr 16 13:59:43.202827 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.202788 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e9f0f164-4788-43c4-890f-52ed0197b594-config\") pod \"service-ca-operator-69965bb79d-hzmxv\" (UID: \"e9f0f164-4788-43c4-890f-52ed0197b594\") " pod="openshift-service-ca-operator/service-ca-operator-69965bb79d-hzmxv" Apr 16 13:59:43.202827 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:59:43.202810 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac-registry-tls podName:eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac nodeName:}" failed. No retries permitted until 2026-04-16 13:59:43.702791245 +0000 UTC m=+153.532526106 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac-registry-tls") pod "image-registry-b76cd7dd-grjcr" (UID: "eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac") : secret "image-registry-tls" not found Apr 16 13:59:43.203030 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.202829 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac-trusted-ca\") pod \"image-registry-b76cd7dd-grjcr\" (UID: \"eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac\") " pod="openshift-image-registry/image-registry-b76cd7dd-grjcr" Apr 16 13:59:43.203030 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.202856 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/5b9fe9ed-1a79-4590-ad6e-74acb8e02aed-telemetry-config\") pod \"cluster-monitoring-operator-6667474d89-tj5v6\" (UID: \"5b9fe9ed-1a79-4590-ad6e-74acb8e02aed\") " pod="openshift-monitoring/cluster-monitoring-operator-6667474d89-tj5v6" Apr 16 13:59:43.203030 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.202904 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5b9fe9ed-1a79-4590-ad6e-74acb8e02aed-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6667474d89-tj5v6\" (UID: \"5b9fe9ed-1a79-4590-ad6e-74acb8e02aed\") " pod="openshift-monitoring/cluster-monitoring-operator-6667474d89-tj5v6" Apr 16 13:59:43.203030 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.203020 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e9f0f164-4788-43c4-890f-52ed0197b594-serving-cert\") pod \"service-ca-operator-69965bb79d-hzmxv\" (UID: \"e9f0f164-4788-43c4-890f-52ed0197b594\") " pod="openshift-service-ca-operator/service-ca-operator-69965bb79d-hzmxv" Apr 16 13:59:43.203193 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.203065 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pdcsf\" (UniqueName: \"kubernetes.io/projected/eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac-kube-api-access-pdcsf\") pod \"image-registry-b76cd7dd-grjcr\" (UID: \"eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac\") " pod="openshift-image-registry/image-registry-b76cd7dd-grjcr" Apr 16 13:59:43.203193 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.203023 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac-registry-certificates\") pod \"image-registry-b76cd7dd-grjcr\" (UID: \"eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac\") " pod="openshift-image-registry/image-registry-b76cd7dd-grjcr" Apr 16 13:59:43.203836 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.203813 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac-trusted-ca\") pod \"image-registry-b76cd7dd-grjcr\" (UID: \"eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac\") " pod="openshift-image-registry/image-registry-b76cd7dd-grjcr" Apr 16 13:59:43.204928 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.204904 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac-installation-pull-secrets\") pod \"image-registry-b76cd7dd-grjcr\" (UID: \"eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac\") " pod="openshift-image-registry/image-registry-b76cd7dd-grjcr" Apr 16 13:59:43.204985 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.204970 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e9f0f164-4788-43c4-890f-52ed0197b594-serving-cert\") pod \"service-ca-operator-69965bb79d-hzmxv\" (UID: \"e9f0f164-4788-43c4-890f-52ed0197b594\") " pod="openshift-service-ca-operator/service-ca-operator-69965bb79d-hzmxv" Apr 16 13:59:43.205145 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.205131 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac-image-registry-private-configuration\") pod \"image-registry-b76cd7dd-grjcr\" (UID: \"eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac\") " pod="openshift-image-registry/image-registry-b76cd7dd-grjcr" Apr 16 13:59:43.211065 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.211041 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac-bound-sa-token\") pod \"image-registry-b76cd7dd-grjcr\" (UID: \"eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac\") " pod="openshift-image-registry/image-registry-b76cd7dd-grjcr" Apr 16 13:59:43.211237 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.211218 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86tzr\" (UniqueName: \"kubernetes.io/projected/e9f0f164-4788-43c4-890f-52ed0197b594-kube-api-access-86tzr\") pod \"service-ca-operator-69965bb79d-hzmxv\" (UID: \"e9f0f164-4788-43c4-890f-52ed0197b594\") " pod="openshift-service-ca-operator/service-ca-operator-69965bb79d-hzmxv" Apr 16 13:59:43.211893 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.211828 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdcsf\" (UniqueName: \"kubernetes.io/projected/eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac-kube-api-access-pdcsf\") pod \"image-registry-b76cd7dd-grjcr\" (UID: \"eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac\") " pod="openshift-image-registry/image-registry-b76cd7dd-grjcr" Apr 16 13:59:43.304252 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.304230 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/5b9fe9ed-1a79-4590-ad6e-74acb8e02aed-telemetry-config\") pod \"cluster-monitoring-operator-6667474d89-tj5v6\" (UID: \"5b9fe9ed-1a79-4590-ad6e-74acb8e02aed\") " pod="openshift-monitoring/cluster-monitoring-operator-6667474d89-tj5v6" Apr 16 13:59:43.304350 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.304265 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5b9fe9ed-1a79-4590-ad6e-74acb8e02aed-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6667474d89-tj5v6\" (UID: \"5b9fe9ed-1a79-4590-ad6e-74acb8e02aed\") " pod="openshift-monitoring/cluster-monitoring-operator-6667474d89-tj5v6" Apr 16 13:59:43.304392 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:59:43.304347 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Apr 16 13:59:43.304392 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:59:43.304391 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b9fe9ed-1a79-4590-ad6e-74acb8e02aed-cluster-monitoring-operator-tls podName:5b9fe9ed-1a79-4590-ad6e-74acb8e02aed nodeName:}" failed. No retries permitted until 2026-04-16 13:59:43.804377761 +0000 UTC m=+153.634112609 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5b9fe9ed-1a79-4590-ad6e-74acb8e02aed-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6667474d89-tj5v6" (UID: "5b9fe9ed-1a79-4590-ad6e-74acb8e02aed") : secret "cluster-monitoring-operator-tls" not found Apr 16 13:59:43.304461 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.304415 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pfdl8\" (UniqueName: \"kubernetes.io/projected/5b9fe9ed-1a79-4590-ad6e-74acb8e02aed-kube-api-access-pfdl8\") pod \"cluster-monitoring-operator-6667474d89-tj5v6\" (UID: \"5b9fe9ed-1a79-4590-ad6e-74acb8e02aed\") " pod="openshift-monitoring/cluster-monitoring-operator-6667474d89-tj5v6" Apr 16 13:59:43.304827 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.304809 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/5b9fe9ed-1a79-4590-ad6e-74acb8e02aed-telemetry-config\") pod \"cluster-monitoring-operator-6667474d89-tj5v6\" (UID: \"5b9fe9ed-1a79-4590-ad6e-74acb8e02aed\") " pod="openshift-monitoring/cluster-monitoring-operator-6667474d89-tj5v6" Apr 16 13:59:43.308511 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.308498 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-69965bb79d-hzmxv" Apr 16 13:59:43.314359 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.314330 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfdl8\" (UniqueName: \"kubernetes.io/projected/5b9fe9ed-1a79-4590-ad6e-74acb8e02aed-kube-api-access-pfdl8\") pod \"cluster-monitoring-operator-6667474d89-tj5v6\" (UID: \"5b9fe9ed-1a79-4590-ad6e-74acb8e02aed\") " pod="openshift-monitoring/cluster-monitoring-operator-6667474d89-tj5v6" Apr 16 13:59:43.423604 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.423576 2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-69965bb79d-hzmxv"] Apr 16 13:59:43.426704 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:59:43.426671 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9f0f164_4788_43c4_890f_52ed0197b594.slice/crio-2efb133d02e2f1b1473b220294ffb392995d34d0b8fa69459aeaa5622b21ce07 WatchSource:0}: Error finding container 2efb133d02e2f1b1473b220294ffb392995d34d0b8fa69459aeaa5622b21ce07: Status 404 returned error can't find the container with id 2efb133d02e2f1b1473b220294ffb392995d34d0b8fa69459aeaa5622b21ce07 Apr 16 13:59:43.606860 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.606831 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e81d5797-d78f-46cf-b792-fdb668a624fc-service-ca-bundle\") pod \"router-default-69cb8777b6-kbr8q\" (UID: \"e81d5797-d78f-46cf-b792-fdb668a624fc\") " pod="openshift-ingress/router-default-69cb8777b6-kbr8q" Apr 16 13:59:43.606959 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.606880 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e81d5797-d78f-46cf-b792-fdb668a624fc-metrics-certs\") pod \"router-default-69cb8777b6-kbr8q\" (UID: \"e81d5797-d78f-46cf-b792-fdb668a624fc\") " pod="openshift-ingress/router-default-69cb8777b6-kbr8q" Apr 16 13:59:43.607013 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:59:43.607001 2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Apr 16 13:59:43.607047 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:59:43.607005 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e81d5797-d78f-46cf-b792-fdb668a624fc-service-ca-bundle podName:e81d5797-d78f-46cf-b792-fdb668a624fc nodeName:}" failed. No retries permitted until 2026-04-16 13:59:44.606988453 +0000 UTC m=+154.436723306 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e81d5797-d78f-46cf-b792-fdb668a624fc-service-ca-bundle") pod "router-default-69cb8777b6-kbr8q" (UID: "e81d5797-d78f-46cf-b792-fdb668a624fc") : configmap references non-existent config key: service-ca.crt Apr 16 13:59:43.607090 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:59:43.607066 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e81d5797-d78f-46cf-b792-fdb668a624fc-metrics-certs podName:e81d5797-d78f-46cf-b792-fdb668a624fc nodeName:}" failed. No retries permitted until 2026-04-16 13:59:44.60704981 +0000 UTC m=+154.436784659 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e81d5797-d78f-46cf-b792-fdb668a624fc-metrics-certs") pod "router-default-69cb8777b6-kbr8q" (UID: "e81d5797-d78f-46cf-b792-fdb668a624fc") : secret "router-metrics-certs-default" not found Apr 16 13:59:43.708084 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.708064 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac-registry-tls\") pod \"image-registry-b76cd7dd-grjcr\" (UID: \"eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac\") " pod="openshift-image-registry/image-registry-b76cd7dd-grjcr" Apr 16 13:59:43.708199 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:59:43.708183 2574 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Apr 16 13:59:43.708243 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:59:43.708200 2574 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b76cd7dd-grjcr: secret "image-registry-tls" not found Apr 16 13:59:43.708243 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:59:43.708237 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac-registry-tls podName:eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac nodeName:}" failed. No retries permitted until 2026-04-16 13:59:44.70822759 +0000 UTC m=+154.537962439 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac-registry-tls") pod "image-registry-b76cd7dd-grjcr" (UID: "eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac") : secret "image-registry-tls" not found Apr 16 13:59:43.808710 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:43.808683 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5b9fe9ed-1a79-4590-ad6e-74acb8e02aed-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6667474d89-tj5v6\" (UID: \"5b9fe9ed-1a79-4590-ad6e-74acb8e02aed\") " pod="openshift-monitoring/cluster-monitoring-operator-6667474d89-tj5v6" Apr 16 13:59:43.808845 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:59:43.808802 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Apr 16 13:59:43.808845 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:59:43.808846 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b9fe9ed-1a79-4590-ad6e-74acb8e02aed-cluster-monitoring-operator-tls podName:5b9fe9ed-1a79-4590-ad6e-74acb8e02aed nodeName:}" failed. No retries permitted until 2026-04-16 13:59:44.808834803 +0000 UTC m=+154.638569652 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5b9fe9ed-1a79-4590-ad6e-74acb8e02aed-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6667474d89-tj5v6" (UID: "5b9fe9ed-1a79-4590-ad6e-74acb8e02aed") : secret "cluster-monitoring-operator-tls" not found Apr 16 13:59:44.162865 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:44.162820 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-69965bb79d-hzmxv" event={"ID":"e9f0f164-4788-43c4-890f-52ed0197b594","Type":"ContainerStarted","Data":"2efb133d02e2f1b1473b220294ffb392995d34d0b8fa69459aeaa5622b21ce07"} Apr 16 13:59:44.614443 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:44.614403 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e81d5797-d78f-46cf-b792-fdb668a624fc-metrics-certs\") pod \"router-default-69cb8777b6-kbr8q\" (UID: \"e81d5797-d78f-46cf-b792-fdb668a624fc\") " pod="openshift-ingress/router-default-69cb8777b6-kbr8q" Apr 16 13:59:44.614600 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:44.614521 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e81d5797-d78f-46cf-b792-fdb668a624fc-service-ca-bundle\") pod \"router-default-69cb8777b6-kbr8q\" (UID: \"e81d5797-d78f-46cf-b792-fdb668a624fc\") " pod="openshift-ingress/router-default-69cb8777b6-kbr8q" Apr 16 13:59:44.614600 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:59:44.614560 2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Apr 16 13:59:44.614688 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:59:44.614630 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e81d5797-d78f-46cf-b792-fdb668a624fc-service-ca-bundle podName:e81d5797-d78f-46cf-b792-fdb668a624fc nodeName:}" failed. No retries permitted until 2026-04-16 13:59:46.614614848 +0000 UTC m=+156.444349698 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e81d5797-d78f-46cf-b792-fdb668a624fc-service-ca-bundle") pod "router-default-69cb8777b6-kbr8q" (UID: "e81d5797-d78f-46cf-b792-fdb668a624fc") : configmap references non-existent config key: service-ca.crt Apr 16 13:59:44.614688 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:59:44.614648 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e81d5797-d78f-46cf-b792-fdb668a624fc-metrics-certs podName:e81d5797-d78f-46cf-b792-fdb668a624fc nodeName:}" failed. No retries permitted until 2026-04-16 13:59:46.614641305 +0000 UTC m=+156.444376153 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e81d5797-d78f-46cf-b792-fdb668a624fc-metrics-certs") pod "router-default-69cb8777b6-kbr8q" (UID: "e81d5797-d78f-46cf-b792-fdb668a624fc") : secret "router-metrics-certs-default" not found Apr 16 13:59:44.715781 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:44.715750 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac-registry-tls\") pod \"image-registry-b76cd7dd-grjcr\" (UID: \"eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac\") " pod="openshift-image-registry/image-registry-b76cd7dd-grjcr" Apr 16 13:59:44.715948 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:59:44.715892 2574 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Apr 16 13:59:44.715948 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:59:44.715913 2574 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b76cd7dd-grjcr: secret "image-registry-tls" not found Apr 16 13:59:44.716048 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:59:44.715968 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac-registry-tls podName:eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac nodeName:}" failed. No retries permitted until 2026-04-16 13:59:46.71595341 +0000 UTC m=+156.545688258 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac-registry-tls") pod "image-registry-b76cd7dd-grjcr" (UID: "eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac") : secret "image-registry-tls" not found Apr 16 13:59:44.817128 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:44.817090 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5b9fe9ed-1a79-4590-ad6e-74acb8e02aed-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6667474d89-tj5v6\" (UID: \"5b9fe9ed-1a79-4590-ad6e-74acb8e02aed\") " pod="openshift-monitoring/cluster-monitoring-operator-6667474d89-tj5v6" Apr 16 13:59:44.817277 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:59:44.817241 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Apr 16 13:59:44.817364 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:59:44.817316 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b9fe9ed-1a79-4590-ad6e-74acb8e02aed-cluster-monitoring-operator-tls podName:5b9fe9ed-1a79-4590-ad6e-74acb8e02aed nodeName:}" failed. No retries permitted until 2026-04-16 13:59:46.817296416 +0000 UTC m=+156.647031269 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5b9fe9ed-1a79-4590-ad6e-74acb8e02aed-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6667474d89-tj5v6" (UID: "5b9fe9ed-1a79-4590-ad6e-74acb8e02aed") : secret "cluster-monitoring-operator-tls" not found Apr 16 13:59:45.166381 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:45.166349 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-69965bb79d-hzmxv" event={"ID":"e9f0f164-4788-43c4-890f-52ed0197b594","Type":"ContainerStarted","Data":"fdfae3918d93a4e1f86251f7f227aff4a3aee07165e13d97032d61e32b18aa1e"} Apr 16 13:59:45.182048 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:45.181997 2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-69965bb79d-hzmxv" podStartSLOduration=1.5465882149999999 podStartE2EDuration="3.181984859s" podCreationTimestamp="2026-04-16 13:59:42 +0000 UTC" firstStartedPulling="2026-04-16 13:59:43.428402382 +0000 UTC m=+153.258137231" lastFinishedPulling="2026-04-16 13:59:45.063799023 +0000 UTC m=+154.893533875" observedRunningTime="2026-04-16 13:59:45.180576978 +0000 UTC m=+155.010311851" watchObservedRunningTime="2026-04-16 13:59:45.181984859 +0000 UTC m=+155.011719728" Apr 16 13:59:46.632056 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:46.632022 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e81d5797-d78f-46cf-b792-fdb668a624fc-service-ca-bundle\") pod \"router-default-69cb8777b6-kbr8q\" (UID: \"e81d5797-d78f-46cf-b792-fdb668a624fc\") " pod="openshift-ingress/router-default-69cb8777b6-kbr8q" Apr 16 13:59:46.632431 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:46.632071 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e81d5797-d78f-46cf-b792-fdb668a624fc-metrics-certs\") pod \"router-default-69cb8777b6-kbr8q\" (UID: \"e81d5797-d78f-46cf-b792-fdb668a624fc\") " pod="openshift-ingress/router-default-69cb8777b6-kbr8q" Apr 16 13:59:46.632431 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:59:46.632174 2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Apr 16 13:59:46.632431 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:59:46.632186 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e81d5797-d78f-46cf-b792-fdb668a624fc-service-ca-bundle podName:e81d5797-d78f-46cf-b792-fdb668a624fc nodeName:}" failed. No retries permitted until 2026-04-16 13:59:50.632165446 +0000 UTC m=+160.461900294 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e81d5797-d78f-46cf-b792-fdb668a624fc-service-ca-bundle") pod "router-default-69cb8777b6-kbr8q" (UID: "e81d5797-d78f-46cf-b792-fdb668a624fc") : configmap references non-existent config key: service-ca.crt Apr 16 13:59:46.632431 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:59:46.632212 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e81d5797-d78f-46cf-b792-fdb668a624fc-metrics-certs podName:e81d5797-d78f-46cf-b792-fdb668a624fc nodeName:}" failed. No retries permitted until 2026-04-16 13:59:50.632200846 +0000 UTC m=+160.461935694 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e81d5797-d78f-46cf-b792-fdb668a624fc-metrics-certs") pod "router-default-69cb8777b6-kbr8q" (UID: "e81d5797-d78f-46cf-b792-fdb668a624fc") : secret "router-metrics-certs-default" not found Apr 16 13:59:46.733016 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:46.732990 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac-registry-tls\") pod \"image-registry-b76cd7dd-grjcr\" (UID: \"eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac\") " pod="openshift-image-registry/image-registry-b76cd7dd-grjcr" Apr 16 13:59:46.733158 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:59:46.733138 2574 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Apr 16 13:59:46.733226 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:59:46.733161 2574 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b76cd7dd-grjcr: secret "image-registry-tls" not found Apr 16 13:59:46.733226 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:59:46.733213 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac-registry-tls podName:eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac nodeName:}" failed. No retries permitted until 2026-04-16 13:59:50.733196905 +0000 UTC m=+160.562931768 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac-registry-tls") pod "image-registry-b76cd7dd-grjcr" (UID: "eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac") : secret "image-registry-tls" not found Apr 16 13:59:46.833971 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:46.833938 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5b9fe9ed-1a79-4590-ad6e-74acb8e02aed-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6667474d89-tj5v6\" (UID: \"5b9fe9ed-1a79-4590-ad6e-74acb8e02aed\") " pod="openshift-monitoring/cluster-monitoring-operator-6667474d89-tj5v6" Apr 16 13:59:46.834084 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:59:46.834031 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Apr 16 13:59:46.834084 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:59:46.834071 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b9fe9ed-1a79-4590-ad6e-74acb8e02aed-cluster-monitoring-operator-tls podName:5b9fe9ed-1a79-4590-ad6e-74acb8e02aed nodeName:}" failed. No retries permitted until 2026-04-16 13:59:50.834059593 +0000 UTC m=+160.663794440 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5b9fe9ed-1a79-4590-ad6e-74acb8e02aed-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6667474d89-tj5v6" (UID: "5b9fe9ed-1a79-4590-ad6e-74acb8e02aed") : secret "cluster-monitoring-operator-tls" not found Apr 16 13:59:48.554236 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:59:48.554178 2574 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-ingress-canary/ingress-canary-xjpgz" podUID="7ee1c9f7-5758-4734-a7b3-ea621cc6df95" Apr 16 13:59:48.558321 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:59:48.558302 2574 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-lnfdm" podUID="eded8e93-5cbc-435c-aab3-a909812ba63b" Apr 16 13:59:48.814499 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:59:48.814422 2574 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-5fh28" podUID="2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e" Apr 16 13:59:49.173144 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:49.173113 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lnfdm" Apr 16 13:59:50.481708 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:50.481680 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-x4qcz_c590b6a7-f5a5-476f-a0bd-fb8e8853b8cb/dns-node-resolver/0.log" Apr 16 13:59:50.663529 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:50.663497 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e81d5797-d78f-46cf-b792-fdb668a624fc-service-ca-bundle\") pod \"router-default-69cb8777b6-kbr8q\" (UID: \"e81d5797-d78f-46cf-b792-fdb668a624fc\") " pod="openshift-ingress/router-default-69cb8777b6-kbr8q" Apr 16 13:59:50.663683 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:50.663553 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e81d5797-d78f-46cf-b792-fdb668a624fc-metrics-certs\") pod \"router-default-69cb8777b6-kbr8q\" (UID: \"e81d5797-d78f-46cf-b792-fdb668a624fc\") " pod="openshift-ingress/router-default-69cb8777b6-kbr8q" Apr 16 13:59:50.663683 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:59:50.663662 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e81d5797-d78f-46cf-b792-fdb668a624fc-service-ca-bundle podName:e81d5797-d78f-46cf-b792-fdb668a624fc nodeName:}" failed. No retries permitted until 2026-04-16 13:59:58.663643663 +0000 UTC m=+168.493378525 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e81d5797-d78f-46cf-b792-fdb668a624fc-service-ca-bundle") pod "router-default-69cb8777b6-kbr8q" (UID: "e81d5797-d78f-46cf-b792-fdb668a624fc") : configmap references non-existent config key: service-ca.crt Apr 16 13:59:50.663683 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:59:50.663680 2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Apr 16 13:59:50.663863 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:59:50.663715 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e81d5797-d78f-46cf-b792-fdb668a624fc-metrics-certs podName:e81d5797-d78f-46cf-b792-fdb668a624fc nodeName:}" failed. No retries permitted until 2026-04-16 13:59:58.663704372 +0000 UTC m=+168.493439234 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e81d5797-d78f-46cf-b792-fdb668a624fc-metrics-certs") pod "router-default-69cb8777b6-kbr8q" (UID: "e81d5797-d78f-46cf-b792-fdb668a624fc") : secret "router-metrics-certs-default" not found Apr 16 13:59:50.764138 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:50.764056 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac-registry-tls\") pod \"image-registry-b76cd7dd-grjcr\" (UID: \"eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac\") " pod="openshift-image-registry/image-registry-b76cd7dd-grjcr" Apr 16 13:59:50.764271 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:59:50.764171 2574 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Apr 16 13:59:50.764271 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:59:50.764181 2574 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b76cd7dd-grjcr: secret "image-registry-tls" not found Apr 16 13:59:50.764271 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:59:50.764225 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac-registry-tls podName:eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac nodeName:}" failed. No retries permitted until 2026-04-16 13:59:58.764212571 +0000 UTC m=+168.593947419 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac-registry-tls") pod "image-registry-b76cd7dd-grjcr" (UID: "eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac") : secret "image-registry-tls" not found Apr 16 13:59:50.864472 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:50.864436 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5b9fe9ed-1a79-4590-ad6e-74acb8e02aed-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6667474d89-tj5v6\" (UID: \"5b9fe9ed-1a79-4590-ad6e-74acb8e02aed\") " pod="openshift-monitoring/cluster-monitoring-operator-6667474d89-tj5v6" Apr 16 13:59:50.864619 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:59:50.864569 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Apr 16 13:59:50.864659 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:59:50.864624 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b9fe9ed-1a79-4590-ad6e-74acb8e02aed-cluster-monitoring-operator-tls podName:5b9fe9ed-1a79-4590-ad6e-74acb8e02aed nodeName:}" failed. No retries permitted until 2026-04-16 13:59:58.864610104 +0000 UTC m=+168.694344956 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5b9fe9ed-1a79-4590-ad6e-74acb8e02aed-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6667474d89-tj5v6" (UID: "5b9fe9ed-1a79-4590-ad6e-74acb8e02aed") : secret "cluster-monitoring-operator-tls" not found Apr 16 13:59:51.480941 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:51.480911 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-brqdj_48194990-0de3-45a9-acc5-6fcc4c2760d0/node-ca/0.log" Apr 16 13:59:53.487191 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:53.487095 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ee1c9f7-5758-4734-a7b3-ea621cc6df95-cert\") pod \"ingress-canary-xjpgz\" (UID: \"7ee1c9f7-5758-4734-a7b3-ea621cc6df95\") " pod="openshift-ingress-canary/ingress-canary-xjpgz" Apr 16 13:59:53.487191 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:53.487143 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eded8e93-5cbc-435c-aab3-a909812ba63b-metrics-tls\") pod \"dns-default-lnfdm\" (UID: \"eded8e93-5cbc-435c-aab3-a909812ba63b\") " pod="openshift-dns/dns-default-lnfdm" Apr 16 13:59:53.487575 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:59:53.487241 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Apr 16 13:59:53.487575 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:59:53.487244 2574 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found Apr 16 13:59:53.487575 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:59:53.487292 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eded8e93-5cbc-435c-aab3-a909812ba63b-metrics-tls podName:eded8e93-5cbc-435c-aab3-a909812ba63b nodeName:}" failed. No retries permitted until 2026-04-16 14:01:55.487277571 +0000 UTC m=+285.317012419 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/eded8e93-5cbc-435c-aab3-a909812ba63b-metrics-tls") pod "dns-default-lnfdm" (UID: "eded8e93-5cbc-435c-aab3-a909812ba63b") : secret "dns-default-metrics-tls" not found Apr 16 13:59:53.487575 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:59:53.487305 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ee1c9f7-5758-4734-a7b3-ea621cc6df95-cert podName:7ee1c9f7-5758-4734-a7b3-ea621cc6df95 nodeName:}" failed. No retries permitted until 2026-04-16 14:01:55.487299659 +0000 UTC m=+285.317034507 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/7ee1c9f7-5758-4734-a7b3-ea621cc6df95-cert") pod "ingress-canary-xjpgz" (UID: "7ee1c9f7-5758-4734-a7b3-ea621cc6df95") : secret "canary-serving-cert" not found Apr 16 13:59:58.725644 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:58.725607 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e81d5797-d78f-46cf-b792-fdb668a624fc-service-ca-bundle\") pod \"router-default-69cb8777b6-kbr8q\" (UID: \"e81d5797-d78f-46cf-b792-fdb668a624fc\") " pod="openshift-ingress/router-default-69cb8777b6-kbr8q" Apr 16 13:59:58.726035 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:58.725665 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e81d5797-d78f-46cf-b792-fdb668a624fc-metrics-certs\") pod \"router-default-69cb8777b6-kbr8q\" (UID: \"e81d5797-d78f-46cf-b792-fdb668a624fc\") " pod="openshift-ingress/router-default-69cb8777b6-kbr8q" Apr 16 13:59:58.726314 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:58.726293 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e81d5797-d78f-46cf-b792-fdb668a624fc-service-ca-bundle\") pod \"router-default-69cb8777b6-kbr8q\" (UID: \"e81d5797-d78f-46cf-b792-fdb668a624fc\") " pod="openshift-ingress/router-default-69cb8777b6-kbr8q" Apr 16 13:59:58.728098 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:58.728074 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e81d5797-d78f-46cf-b792-fdb668a624fc-metrics-certs\") pod \"router-default-69cb8777b6-kbr8q\" (UID: \"e81d5797-d78f-46cf-b792-fdb668a624fc\") " pod="openshift-ingress/router-default-69cb8777b6-kbr8q" Apr 16 13:59:58.800089 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:58.800063 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-69cb8777b6-kbr8q" Apr 16 13:59:58.826881 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:58.826854 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac-registry-tls\") pod \"image-registry-b76cd7dd-grjcr\" (UID: \"eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac\") " pod="openshift-image-registry/image-registry-b76cd7dd-grjcr" Apr 16 13:59:58.829098 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:58.829069 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac-registry-tls\") pod \"image-registry-b76cd7dd-grjcr\" (UID: \"eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac\") " pod="openshift-image-registry/image-registry-b76cd7dd-grjcr" Apr 16 13:59:58.904141 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:58.904110 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b76cd7dd-grjcr" Apr 16 13:59:58.913866 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:58.913842 2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-69cb8777b6-kbr8q"] Apr 16 13:59:58.916230 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:59:58.916200 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode81d5797_d78f_46cf_b792_fdb668a624fc.slice/crio-397e598ee465f5b0504407c97e84209a32d41d17e637f45e11cf021bc6c7b0e3 WatchSource:0}: Error finding container 397e598ee465f5b0504407c97e84209a32d41d17e637f45e11cf021bc6c7b0e3: Status 404 returned error can't find the container with id 397e598ee465f5b0504407c97e84209a32d41d17e637f45e11cf021bc6c7b0e3 Apr 16 13:59:58.928283 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:58.928253 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5b9fe9ed-1a79-4590-ad6e-74acb8e02aed-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6667474d89-tj5v6\" (UID: \"5b9fe9ed-1a79-4590-ad6e-74acb8e02aed\") " pod="openshift-monitoring/cluster-monitoring-operator-6667474d89-tj5v6" Apr 16 13:59:58.928398 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:59:58.928381 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Apr 16 13:59:58.928464 ip-10-0-136-114 kubenswrapper[2574]: E0416 13:59:58.928454 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b9fe9ed-1a79-4590-ad6e-74acb8e02aed-cluster-monitoring-operator-tls podName:5b9fe9ed-1a79-4590-ad6e-74acb8e02aed nodeName:}" failed. No retries permitted until 2026-04-16 14:00:14.928424637 +0000 UTC m=+184.758159485 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5b9fe9ed-1a79-4590-ad6e-74acb8e02aed-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6667474d89-tj5v6" (UID: "5b9fe9ed-1a79-4590-ad6e-74acb8e02aed") : secret "cluster-monitoring-operator-tls" not found Apr 16 13:59:59.028804 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:59.028775 2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-b76cd7dd-grjcr"] Apr 16 13:59:59.031566 ip-10-0-136-114 kubenswrapper[2574]: W0416 13:59:59.031543 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb098e1c_6a7b_4d8b_a47e_91a6a1a2f5ac.slice/crio-8c0ee2c300c34de93230429ffafcc2f3220b46ecbb2108dc0209423829ab7913 WatchSource:0}: Error finding container 8c0ee2c300c34de93230429ffafcc2f3220b46ecbb2108dc0209423829ab7913: Status 404 returned error can't find the container with id 8c0ee2c300c34de93230429ffafcc2f3220b46ecbb2108dc0209423829ab7913 Apr 16 13:59:59.195663 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:59.195623 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b76cd7dd-grjcr" event={"ID":"eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac","Type":"ContainerStarted","Data":"3324f8d8af0df86b61f76ef9abb9f8496e75681cb3fd39053fe38fd20bf5271f"} Apr 16 13:59:59.195663 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:59.195663 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b76cd7dd-grjcr" event={"ID":"eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac","Type":"ContainerStarted","Data":"8c0ee2c300c34de93230429ffafcc2f3220b46ecbb2108dc0209423829ab7913"} Apr 16 13:59:59.196020 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:59.195757 2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-b76cd7dd-grjcr" Apr 16 13:59:59.196946 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:59.196916 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-69cb8777b6-kbr8q" event={"ID":"e81d5797-d78f-46cf-b792-fdb668a624fc","Type":"ContainerStarted","Data":"145017bbf5957bd15a216453dd8c7093957212b65b3f6a2c2b374ad9ab962144"} Apr 16 13:59:59.196946 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:59.196945 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-69cb8777b6-kbr8q" event={"ID":"e81d5797-d78f-46cf-b792-fdb668a624fc","Type":"ContainerStarted","Data":"397e598ee465f5b0504407c97e84209a32d41d17e637f45e11cf021bc6c7b0e3"} Apr 16 13:59:59.215130 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:59.215073 2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/image-registry-b76cd7dd-grjcr" podStartSLOduration=17.215055993 podStartE2EDuration="17.215055993s" podCreationTimestamp="2026-04-16 13:59:42 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 13:59:59.214329509 +0000 UTC m=+169.044064379" watchObservedRunningTime="2026-04-16 13:59:59.215055993 +0000 UTC m=+169.044790861" Apr 16 13:59:59.233314 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:59.233249 2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress/router-default-69cb8777b6-kbr8q" podStartSLOduration=17.233233537 podStartE2EDuration="17.233233537s" podCreationTimestamp="2026-04-16 13:59:42 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 13:59:59.23268678 +0000 UTC m=+169.062421651" watchObservedRunningTime="2026-04-16 13:59:59.233233537 +0000 UTC m=+169.062968408" Apr 16 13:59:59.801202 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:59.801177 2574 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-69cb8777b6-kbr8q" Apr 16 13:59:59.803513 ip-10-0-136-114 kubenswrapper[2574]: I0416 13:59:59.803492 2574 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-69cb8777b6-kbr8q" Apr 16 14:00:00.201245 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:00.201206 2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-69cb8777b6-kbr8q" Apr 16 14:00:00.202279 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:00.202259 2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-69cb8777b6-kbr8q" Apr 16 14:00:00.806015 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:00.805976 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xjpgz" Apr 16 14:00:01.805369 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:01.805317 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5fh28" Apr 16 14:00:11.324878 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:11.324840 2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-b76cd7dd-grjcr"] Apr 16 14:00:11.409468 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:11.409438 2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-wdl6d"] Apr 16 14:00:11.413280 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:11.413259 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wdl6d" Apr 16 14:00:11.416842 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:11.416818 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Apr 16 14:00:11.416952 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:11.416869 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Apr 16 14:00:11.416952 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:11.416918 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Apr 16 14:00:11.417065 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:11.416952 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Apr 16 14:00:11.417295 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:11.417281 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vbnth\"" Apr 16 14:00:11.434879 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:11.434855 2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wdl6d"] Apr 16 14:00:11.526075 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:11.526038 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ff5a884a-7809-4fea-a804-78ce4afaa27e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wdl6d\" (UID: \"ff5a884a-7809-4fea-a804-78ce4afaa27e\") " pod="openshift-insights/insights-runtime-extractor-wdl6d" Apr 16 14:00:11.526255 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:11.526081 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ff5a884a-7809-4fea-a804-78ce4afaa27e-crio-socket\") pod \"insights-runtime-extractor-wdl6d\" (UID: \"ff5a884a-7809-4fea-a804-78ce4afaa27e\") " pod="openshift-insights/insights-runtime-extractor-wdl6d" Apr 16 14:00:11.526255 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:11.526134 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ff5a884a-7809-4fea-a804-78ce4afaa27e-data-volume\") pod \"insights-runtime-extractor-wdl6d\" (UID: \"ff5a884a-7809-4fea-a804-78ce4afaa27e\") " pod="openshift-insights/insights-runtime-extractor-wdl6d" Apr 16 14:00:11.526255 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:11.526158 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ff5a884a-7809-4fea-a804-78ce4afaa27e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wdl6d\" (UID: \"ff5a884a-7809-4fea-a804-78ce4afaa27e\") " pod="openshift-insights/insights-runtime-extractor-wdl6d" Apr 16 14:00:11.526255 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:11.526242 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjwkx\" (UniqueName: \"kubernetes.io/projected/ff5a884a-7809-4fea-a804-78ce4afaa27e-kube-api-access-cjwkx\") pod \"insights-runtime-extractor-wdl6d\" (UID: \"ff5a884a-7809-4fea-a804-78ce4afaa27e\") " pod="openshift-insights/insights-runtime-extractor-wdl6d" Apr 16 14:00:11.627092 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:11.627062 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ff5a884a-7809-4fea-a804-78ce4afaa27e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wdl6d\" (UID: \"ff5a884a-7809-4fea-a804-78ce4afaa27e\") " pod="openshift-insights/insights-runtime-extractor-wdl6d" Apr 16 14:00:11.627255 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:11.627101 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ff5a884a-7809-4fea-a804-78ce4afaa27e-crio-socket\") pod \"insights-runtime-extractor-wdl6d\" (UID: \"ff5a884a-7809-4fea-a804-78ce4afaa27e\") " pod="openshift-insights/insights-runtime-extractor-wdl6d" Apr 16 14:00:11.627255 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:11.627150 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ff5a884a-7809-4fea-a804-78ce4afaa27e-data-volume\") pod \"insights-runtime-extractor-wdl6d\" (UID: \"ff5a884a-7809-4fea-a804-78ce4afaa27e\") " pod="openshift-insights/insights-runtime-extractor-wdl6d" Apr 16 14:00:11.627255 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:11.627172 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ff5a884a-7809-4fea-a804-78ce4afaa27e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wdl6d\" (UID: \"ff5a884a-7809-4fea-a804-78ce4afaa27e\") " pod="openshift-insights/insights-runtime-extractor-wdl6d" Apr 16 14:00:11.627255 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:11.627213 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-cjwkx\" (UniqueName: \"kubernetes.io/projected/ff5a884a-7809-4fea-a804-78ce4afaa27e-kube-api-access-cjwkx\") pod \"insights-runtime-extractor-wdl6d\" (UID: \"ff5a884a-7809-4fea-a804-78ce4afaa27e\") " pod="openshift-insights/insights-runtime-extractor-wdl6d" Apr 16 14:00:11.627446 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:11.627256 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ff5a884a-7809-4fea-a804-78ce4afaa27e-crio-socket\") pod \"insights-runtime-extractor-wdl6d\" (UID: \"ff5a884a-7809-4fea-a804-78ce4afaa27e\") " pod="openshift-insights/insights-runtime-extractor-wdl6d" Apr 16 14:00:11.627553 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:11.627532 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ff5a884a-7809-4fea-a804-78ce4afaa27e-data-volume\") pod \"insights-runtime-extractor-wdl6d\" (UID: \"ff5a884a-7809-4fea-a804-78ce4afaa27e\") " pod="openshift-insights/insights-runtime-extractor-wdl6d" Apr 16 14:00:11.627665 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:11.627649 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ff5a884a-7809-4fea-a804-78ce4afaa27e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wdl6d\" (UID: \"ff5a884a-7809-4fea-a804-78ce4afaa27e\") " pod="openshift-insights/insights-runtime-extractor-wdl6d" Apr 16 14:00:11.629302 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:11.629284 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ff5a884a-7809-4fea-a804-78ce4afaa27e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wdl6d\" (UID: \"ff5a884a-7809-4fea-a804-78ce4afaa27e\") " pod="openshift-insights/insights-runtime-extractor-wdl6d" Apr 16 14:00:11.638392 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:11.638369 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjwkx\" (UniqueName: \"kubernetes.io/projected/ff5a884a-7809-4fea-a804-78ce4afaa27e-kube-api-access-cjwkx\") pod \"insights-runtime-extractor-wdl6d\" (UID: \"ff5a884a-7809-4fea-a804-78ce4afaa27e\") " pod="openshift-insights/insights-runtime-extractor-wdl6d" Apr 16 14:00:11.723298 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:11.723282 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wdl6d" Apr 16 14:00:11.843266 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:11.843236 2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wdl6d"] Apr 16 14:00:11.846327 ip-10-0-136-114 kubenswrapper[2574]: W0416 14:00:11.846303 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff5a884a_7809_4fea_a804_78ce4afaa27e.slice/crio-70db4d8efe9d0b0710897ddbfcf4e6382e3c4242cd286217e41c1943c457a744 WatchSource:0}: Error finding container 70db4d8efe9d0b0710897ddbfcf4e6382e3c4242cd286217e41c1943c457a744: Status 404 returned error can't find the container with id 70db4d8efe9d0b0710897ddbfcf4e6382e3c4242cd286217e41c1943c457a744 Apr 16 14:00:12.227806 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:12.227711 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wdl6d" event={"ID":"ff5a884a-7809-4fea-a804-78ce4afaa27e","Type":"ContainerStarted","Data":"bab5658ef0b84f3a9413519fec0072db1c32caa42bff8c86e4e29ce6336bb579"} Apr 16 14:00:12.227806 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:12.227763 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wdl6d" event={"ID":"ff5a884a-7809-4fea-a804-78ce4afaa27e","Type":"ContainerStarted","Data":"70db4d8efe9d0b0710897ddbfcf4e6382e3c4242cd286217e41c1943c457a744"} Apr 16 14:00:13.230823 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:13.230786 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wdl6d" event={"ID":"ff5a884a-7809-4fea-a804-78ce4afaa27e","Type":"ContainerStarted","Data":"939f690da6fcad01c5de07f48e518bd57a6dea5259c9a902006bc32c4c42c75f"} Apr 16 14:00:14.953640 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:14.953607 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5b9fe9ed-1a79-4590-ad6e-74acb8e02aed-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6667474d89-tj5v6\" (UID: \"5b9fe9ed-1a79-4590-ad6e-74acb8e02aed\") " pod="openshift-monitoring/cluster-monitoring-operator-6667474d89-tj5v6" Apr 16 14:00:14.956103 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:14.956078 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5b9fe9ed-1a79-4590-ad6e-74acb8e02aed-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6667474d89-tj5v6\" (UID: \"5b9fe9ed-1a79-4590-ad6e-74acb8e02aed\") " pod="openshift-monitoring/cluster-monitoring-operator-6667474d89-tj5v6" Apr 16 14:00:15.197012 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:15.196980 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-x6sn6\"" Apr 16 14:00:15.205159 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:15.205112 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6667474d89-tj5v6" Apr 16 14:00:15.242712 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:15.242654 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wdl6d" event={"ID":"ff5a884a-7809-4fea-a804-78ce4afaa27e","Type":"ContainerStarted","Data":"755b59a0b7eaf3e73782228076d0c22b2d611c8c12b379a123eba690f7bc5fd6"} Apr 16 14:00:15.263189 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:15.263137 2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-wdl6d" podStartSLOduration=1.175221802 podStartE2EDuration="4.263124118s" podCreationTimestamp="2026-04-16 14:00:11 +0000 UTC" firstStartedPulling="2026-04-16 14:00:11.89503377 +0000 UTC m=+181.724768618" lastFinishedPulling="2026-04-16 14:00:14.982936085 +0000 UTC m=+184.812670934" observedRunningTime="2026-04-16 14:00:15.262389779 +0000 UTC m=+185.092124663" watchObservedRunningTime="2026-04-16 14:00:15.263124118 +0000 UTC m=+185.092858988" Apr 16 14:00:15.319233 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:15.319206 2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6667474d89-tj5v6"] Apr 16 14:00:15.322504 ip-10-0-136-114 kubenswrapper[2574]: W0416 14:00:15.322478 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b9fe9ed_1a79_4590_ad6e_74acb8e02aed.slice/crio-c63e6becf9ffea8c9e56beb29c11dd448420df9382105a8c068ecb8ad97a47a6 WatchSource:0}: Error finding container c63e6becf9ffea8c9e56beb29c11dd448420df9382105a8c068ecb8ad97a47a6: Status 404 returned error can't find the container with id c63e6becf9ffea8c9e56beb29c11dd448420df9382105a8c068ecb8ad97a47a6 Apr 16 14:00:16.246168 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:16.246127 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6667474d89-tj5v6" event={"ID":"5b9fe9ed-1a79-4590-ad6e-74acb8e02aed","Type":"ContainerStarted","Data":"c63e6becf9ffea8c9e56beb29c11dd448420df9382105a8c068ecb8ad97a47a6"} Apr 16 14:00:17.250305 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:17.250270 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6667474d89-tj5v6" event={"ID":"5b9fe9ed-1a79-4590-ad6e-74acb8e02aed","Type":"ContainerStarted","Data":"1d8338b5ca3154b7fae96c1a22402a026c5a00ac22ac996a2f97b419aadaa8c0"} Apr 16 14:00:17.267333 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:17.267277 2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6667474d89-tj5v6" podStartSLOduration=32.520975338 podStartE2EDuration="34.267263637s" podCreationTimestamp="2026-04-16 13:59:43 +0000 UTC" firstStartedPulling="2026-04-16 14:00:15.324222739 +0000 UTC m=+185.153957590" lastFinishedPulling="2026-04-16 14:00:17.070511042 +0000 UTC m=+186.900245889" observedRunningTime="2026-04-16 14:00:17.265623576 +0000 UTC m=+187.095358446" watchObservedRunningTime="2026-04-16 14:00:17.267263637 +0000 UTC m=+187.096998566" Apr 16 14:00:17.584761 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:17.584714 2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-9cb97cd87-cnfb5"] Apr 16 14:00:17.587421 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:17.587405 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-9cb97cd87-cnfb5" Apr 16 14:00:17.590322 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:17.590297 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Apr 16 14:00:17.590427 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:17.590353 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-jrdm9\"" Apr 16 14:00:17.599506 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:17.599487 2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-9cb97cd87-cnfb5"] Apr 16 14:00:17.675696 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:17.675665 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4f118646-7573-4184-880d-eb313a7b063d-tls-certificates\") pod \"prometheus-operator-admission-webhook-9cb97cd87-cnfb5\" (UID: \"4f118646-7573-4184-880d-eb313a7b063d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-9cb97cd87-cnfb5" Apr 16 14:00:17.776824 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:17.776788 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4f118646-7573-4184-880d-eb313a7b063d-tls-certificates\") pod \"prometheus-operator-admission-webhook-9cb97cd87-cnfb5\" (UID: \"4f118646-7573-4184-880d-eb313a7b063d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-9cb97cd87-cnfb5" Apr 16 14:00:17.776969 ip-10-0-136-114 kubenswrapper[2574]: E0416 14:00:17.776949 2574 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Apr 16 14:00:17.777051 ip-10-0-136-114 kubenswrapper[2574]: E0416 14:00:17.777040 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f118646-7573-4184-880d-eb313a7b063d-tls-certificates podName:4f118646-7573-4184-880d-eb313a7b063d nodeName:}" failed. No retries permitted until 2026-04-16 14:00:18.277020272 +0000 UTC m=+188.106755123 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/4f118646-7573-4184-880d-eb313a7b063d-tls-certificates") pod "prometheus-operator-admission-webhook-9cb97cd87-cnfb5" (UID: "4f118646-7573-4184-880d-eb313a7b063d") : secret "prometheus-operator-admission-webhook-tls" not found Apr 16 14:00:18.280388 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:18.280353 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4f118646-7573-4184-880d-eb313a7b063d-tls-certificates\") pod \"prometheus-operator-admission-webhook-9cb97cd87-cnfb5\" (UID: \"4f118646-7573-4184-880d-eb313a7b063d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-9cb97cd87-cnfb5" Apr 16 14:00:18.283113 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:18.283087 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4f118646-7573-4184-880d-eb313a7b063d-tls-certificates\") pod \"prometheus-operator-admission-webhook-9cb97cd87-cnfb5\" (UID: \"4f118646-7573-4184-880d-eb313a7b063d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-9cb97cd87-cnfb5" Apr 16 14:00:18.496222 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:18.496193 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-9cb97cd87-cnfb5" Apr 16 14:00:18.609499 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:18.609471 2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-9cb97cd87-cnfb5"] Apr 16 14:00:18.612585 ip-10-0-136-114 kubenswrapper[2574]: W0416 14:00:18.612556 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f118646_7573_4184_880d_eb313a7b063d.slice/crio-4ed7c3e8712b516de5563e172a2713274b399cb6651bfd4988a9c7b578df78f2 WatchSource:0}: Error finding container 4ed7c3e8712b516de5563e172a2713274b399cb6651bfd4988a9c7b578df78f2: Status 404 returned error can't find the container with id 4ed7c3e8712b516de5563e172a2713274b399cb6651bfd4988a9c7b578df78f2 Apr 16 14:00:19.255996 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:19.255954 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-9cb97cd87-cnfb5" event={"ID":"4f118646-7573-4184-880d-eb313a7b063d","Type":"ContainerStarted","Data":"4ed7c3e8712b516de5563e172a2713274b399cb6651bfd4988a9c7b578df78f2"} Apr 16 14:00:20.260929 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:20.260892 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-9cb97cd87-cnfb5" event={"ID":"4f118646-7573-4184-880d-eb313a7b063d","Type":"ContainerStarted","Data":"1ba03e07257393b113ce6394855c1188c70b0e242f38a9fc968264a18ec2c709"} Apr 16 14:00:20.261309 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:20.261087 2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-9cb97cd87-cnfb5" Apr 16 14:00:20.265625 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:20.265588 2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-9cb97cd87-cnfb5" Apr 16 14:00:20.301587 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:20.301495 2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-9cb97cd87-cnfb5" podStartSLOduration=1.872446917 podStartE2EDuration="3.301482337s" podCreationTimestamp="2026-04-16 14:00:17 +0000 UTC" firstStartedPulling="2026-04-16 14:00:18.61454147 +0000 UTC m=+188.444276317" lastFinishedPulling="2026-04-16 14:00:20.043576889 +0000 UTC m=+189.873311737" observedRunningTime="2026-04-16 14:00:20.28355452 +0000 UTC m=+190.113289392" watchObservedRunningTime="2026-04-16 14:00:20.301482337 +0000 UTC m=+190.131217209" Apr 16 14:00:21.329355 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:21.329332 2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-b76cd7dd-grjcr" Apr 16 14:00:25.983683 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:25.983649 2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-5669946b84-q57gd"] Apr 16 14:00:25.986759 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:25.986728 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-5669946b84-q57gd" Apr 16 14:00:25.989167 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:25.989147 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Apr 16 14:00:25.990127 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:25.990108 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Apr 16 14:00:25.990127 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:25.990115 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-7qm67\"" Apr 16 14:00:25.990250 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:25.990221 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Apr 16 14:00:25.998254 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:25.998225 2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7479c89684-8xh94"] Apr 16 14:00:26.004995 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.004970 2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-5669946b84-q57gd"] Apr 16 14:00:26.005096 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.005001 2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-h7mfj"] Apr 16 14:00:26.005154 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.005116 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7479c89684-8xh94" Apr 16 14:00:26.007665 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.007637 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Apr 16 14:00:26.007866 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.007851 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-h7mh8\"" Apr 16 14:00:26.007937 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.007896 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Apr 16 14:00:26.007937 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.007927 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Apr 16 14:00:26.008216 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.008201 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-h7mfj" Apr 16 14:00:26.010127 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.010109 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Apr 16 14:00:26.010192 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.010165 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2gbx5\"" Apr 16 14:00:26.010192 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.010184 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Apr 16 14:00:26.010362 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.010348 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Apr 16 14:00:26.017286 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.017263 2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7479c89684-8xh94"] Apr 16 14:00:26.038202 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.038176 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq6bk\" (UniqueName: \"kubernetes.io/projected/0d969860-3250-40b5-b124-a01d0c99a8a1-kube-api-access-pq6bk\") pod \"openshift-state-metrics-5669946b84-q57gd\" (UID: \"0d969860-3250-40b5-b124-a01d0c99a8a1\") " pod="openshift-monitoring/openshift-state-metrics-5669946b84-q57gd" Apr 16 14:00:26.038297 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.038224 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d969860-3250-40b5-b124-a01d0c99a8a1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-5669946b84-q57gd\" (UID: \"0d969860-3250-40b5-b124-a01d0c99a8a1\") " pod="openshift-monitoring/openshift-state-metrics-5669946b84-q57gd" Apr 16 14:00:26.038297 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.038243 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0d969860-3250-40b5-b124-a01d0c99a8a1-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-5669946b84-q57gd\" (UID: \"0d969860-3250-40b5-b124-a01d0c99a8a1\") " pod="openshift-monitoring/openshift-state-metrics-5669946b84-q57gd" Apr 16 14:00:26.038297 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.038292 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d969860-3250-40b5-b124-a01d0c99a8a1-metrics-client-ca\") pod \"openshift-state-metrics-5669946b84-q57gd\" (UID: \"0d969860-3250-40b5-b124-a01d0c99a8a1\") " pod="openshift-monitoring/openshift-state-metrics-5669946b84-q57gd" Apr 16 14:00:26.139344 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.139310 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qmhl\" (UniqueName: \"kubernetes.io/projected/35436df9-1548-4b04-9e65-ca60b80486e7-kube-api-access-8qmhl\") pod \"node-exporter-h7mfj\" (UID: \"35436df9-1548-4b04-9e65-ca60b80486e7\") " pod="openshift-monitoring/node-exporter-h7mfj" Apr 16 14:00:26.139456 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.139350 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74619757-38ff-48ef-9c0c-129c39811a4d-metrics-client-ca\") pod \"kube-state-metrics-7479c89684-8xh94\" (UID: \"74619757-38ff-48ef-9c0c-129c39811a4d\") " pod="openshift-monitoring/kube-state-metrics-7479c89684-8xh94" Apr 16 14:00:26.139456 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.139369 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/35436df9-1548-4b04-9e65-ca60b80486e7-node-exporter-accelerators-collector-config\") pod \"node-exporter-h7mfj\" (UID: \"35436df9-1548-4b04-9e65-ca60b80486e7\") " pod="openshift-monitoring/node-exporter-h7mfj" Apr 16 14:00:26.139456 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.139435 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fj2r\" (UniqueName: \"kubernetes.io/projected/74619757-38ff-48ef-9c0c-129c39811a4d-kube-api-access-7fj2r\") pod \"kube-state-metrics-7479c89684-8xh94\" (UID: \"74619757-38ff-48ef-9c0c-129c39811a4d\") " pod="openshift-monitoring/kube-state-metrics-7479c89684-8xh94" Apr 16 14:00:26.139566 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.139468 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/74619757-38ff-48ef-9c0c-129c39811a4d-kube-state-metrics-tls\") pod \"kube-state-metrics-7479c89684-8xh94\" (UID: \"74619757-38ff-48ef-9c0c-129c39811a4d\") " pod="openshift-monitoring/kube-state-metrics-7479c89684-8xh94" Apr 16 14:00:26.139566 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.139486 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/35436df9-1548-4b04-9e65-ca60b80486e7-node-exporter-tls\") pod \"node-exporter-h7mfj\" (UID: \"35436df9-1548-4b04-9e65-ca60b80486e7\") " pod="openshift-monitoring/node-exporter-h7mfj" Apr 16 14:00:26.139566 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.139526 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/35436df9-1548-4b04-9e65-ca60b80486e7-sys\") pod \"node-exporter-h7mfj\" (UID: \"35436df9-1548-4b04-9e65-ca60b80486e7\") " pod="openshift-monitoring/node-exporter-h7mfj" Apr 16 14:00:26.139566 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.139547 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pq6bk\" (UniqueName: \"kubernetes.io/projected/0d969860-3250-40b5-b124-a01d0c99a8a1-kube-api-access-pq6bk\") pod \"openshift-state-metrics-5669946b84-q57gd\" (UID: \"0d969860-3250-40b5-b124-a01d0c99a8a1\") " pod="openshift-monitoring/openshift-state-metrics-5669946b84-q57gd" Apr 16 14:00:26.139682 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.139583 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/74619757-38ff-48ef-9c0c-129c39811a4d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7479c89684-8xh94\" (UID: \"74619757-38ff-48ef-9c0c-129c39811a4d\") " pod="openshift-monitoring/kube-state-metrics-7479c89684-8xh94" Apr 16 14:00:26.139682 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.139616 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/74619757-38ff-48ef-9c0c-129c39811a4d-volume-directive-shadow\") pod \"kube-state-metrics-7479c89684-8xh94\" (UID: \"74619757-38ff-48ef-9c0c-129c39811a4d\") " pod="openshift-monitoring/kube-state-metrics-7479c89684-8xh94" Apr 16 14:00:26.139682 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.139638 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/35436df9-1548-4b04-9e65-ca60b80486e7-root\") pod \"node-exporter-h7mfj\" (UID: \"35436df9-1548-4b04-9e65-ca60b80486e7\") " pod="openshift-monitoring/node-exporter-h7mfj" Apr 16 14:00:26.139682 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.139654 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/35436df9-1548-4b04-9e65-ca60b80486e7-node-exporter-textfile\") pod \"node-exporter-h7mfj\" (UID: \"35436df9-1548-4b04-9e65-ca60b80486e7\") " pod="openshift-monitoring/node-exporter-h7mfj" Apr 16 14:00:26.139682 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.139670 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/74619757-38ff-48ef-9c0c-129c39811a4d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7479c89684-8xh94\" (UID: \"74619757-38ff-48ef-9c0c-129c39811a4d\") " pod="openshift-monitoring/kube-state-metrics-7479c89684-8xh94" Apr 16 14:00:26.139923 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.139706 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d969860-3250-40b5-b124-a01d0c99a8a1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-5669946b84-q57gd\" (UID: \"0d969860-3250-40b5-b124-a01d0c99a8a1\") " pod="openshift-monitoring/openshift-state-metrics-5669946b84-q57gd" Apr 16 14:00:26.139923 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.139728 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0d969860-3250-40b5-b124-a01d0c99a8a1-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-5669946b84-q57gd\" (UID: \"0d969860-3250-40b5-b124-a01d0c99a8a1\") " pod="openshift-monitoring/openshift-state-metrics-5669946b84-q57gd" Apr 16 14:00:26.139923 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.139759 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/35436df9-1548-4b04-9e65-ca60b80486e7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h7mfj\" (UID: \"35436df9-1548-4b04-9e65-ca60b80486e7\") " pod="openshift-monitoring/node-exporter-h7mfj" Apr 16 14:00:26.139923 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.139788 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35436df9-1548-4b04-9e65-ca60b80486e7-metrics-client-ca\") pod \"node-exporter-h7mfj\" (UID: \"35436df9-1548-4b04-9e65-ca60b80486e7\") " pod="openshift-monitoring/node-exporter-h7mfj" Apr 16 14:00:26.139923 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.139846 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d969860-3250-40b5-b124-a01d0c99a8a1-metrics-client-ca\") pod \"openshift-state-metrics-5669946b84-q57gd\" (UID: \"0d969860-3250-40b5-b124-a01d0c99a8a1\") " pod="openshift-monitoring/openshift-state-metrics-5669946b84-q57gd" Apr 16 14:00:26.139923 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.139891 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/35436df9-1548-4b04-9e65-ca60b80486e7-node-exporter-wtmp\") pod \"node-exporter-h7mfj\" (UID: \"35436df9-1548-4b04-9e65-ca60b80486e7\") " pod="openshift-monitoring/node-exporter-h7mfj" Apr 16 14:00:26.139923 ip-10-0-136-114 kubenswrapper[2574]: E0416 14:00:26.139855 2574 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Apr 16 14:00:26.140161 ip-10-0-136-114 kubenswrapper[2574]: E0416 14:00:26.139972 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d969860-3250-40b5-b124-a01d0c99a8a1-openshift-state-metrics-tls podName:0d969860-3250-40b5-b124-a01d0c99a8a1 nodeName:}" failed. No retries permitted until 2026-04-16 14:00:26.639957046 +0000 UTC m=+196.469691894 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/0d969860-3250-40b5-b124-a01d0c99a8a1-openshift-state-metrics-tls") pod "openshift-state-metrics-5669946b84-q57gd" (UID: "0d969860-3250-40b5-b124-a01d0c99a8a1") : secret "openshift-state-metrics-tls" not found Apr 16 14:00:26.140399 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.140383 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d969860-3250-40b5-b124-a01d0c99a8a1-metrics-client-ca\") pod \"openshift-state-metrics-5669946b84-q57gd\" (UID: \"0d969860-3250-40b5-b124-a01d0c99a8a1\") " pod="openshift-monitoring/openshift-state-metrics-5669946b84-q57gd" Apr 16 14:00:26.141935 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.141918 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0d969860-3250-40b5-b124-a01d0c99a8a1-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-5669946b84-q57gd\" (UID: \"0d969860-3250-40b5-b124-a01d0c99a8a1\") " pod="openshift-monitoring/openshift-state-metrics-5669946b84-q57gd" Apr 16 14:00:26.150086 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.150059 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq6bk\" (UniqueName: \"kubernetes.io/projected/0d969860-3250-40b5-b124-a01d0c99a8a1-kube-api-access-pq6bk\") pod \"openshift-state-metrics-5669946b84-q57gd\" (UID: \"0d969860-3250-40b5-b124-a01d0c99a8a1\") " pod="openshift-monitoring/openshift-state-metrics-5669946b84-q57gd" Apr 16 14:00:26.240757 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.240687 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/74619757-38ff-48ef-9c0c-129c39811a4d-volume-directive-shadow\") pod \"kube-state-metrics-7479c89684-8xh94\" (UID: \"74619757-38ff-48ef-9c0c-129c39811a4d\") " pod="openshift-monitoring/kube-state-metrics-7479c89684-8xh94" Apr 16 14:00:26.240757 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.240716 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/35436df9-1548-4b04-9e65-ca60b80486e7-root\") pod \"node-exporter-h7mfj\" (UID: \"35436df9-1548-4b04-9e65-ca60b80486e7\") " pod="openshift-monitoring/node-exporter-h7mfj" Apr 16 14:00:26.240757 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.240734 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/35436df9-1548-4b04-9e65-ca60b80486e7-node-exporter-textfile\") pod \"node-exporter-h7mfj\" (UID: \"35436df9-1548-4b04-9e65-ca60b80486e7\") " pod="openshift-monitoring/node-exporter-h7mfj" Apr 16 14:00:26.240978 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.240774 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/74619757-38ff-48ef-9c0c-129c39811a4d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7479c89684-8xh94\" (UID: \"74619757-38ff-48ef-9c0c-129c39811a4d\") " pod="openshift-monitoring/kube-state-metrics-7479c89684-8xh94" Apr 16 14:00:26.240978 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.240816 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/35436df9-1548-4b04-9e65-ca60b80486e7-root\") pod \"node-exporter-h7mfj\" (UID: \"35436df9-1548-4b04-9e65-ca60b80486e7\") " pod="openshift-monitoring/node-exporter-h7mfj" Apr 16 14:00:26.240978 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.240859 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/35436df9-1548-4b04-9e65-ca60b80486e7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h7mfj\" (UID: \"35436df9-1548-4b04-9e65-ca60b80486e7\") " pod="openshift-monitoring/node-exporter-h7mfj" Apr 16 14:00:26.240978 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.240892 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35436df9-1548-4b04-9e65-ca60b80486e7-metrics-client-ca\") pod \"node-exporter-h7mfj\" (UID: \"35436df9-1548-4b04-9e65-ca60b80486e7\") " pod="openshift-monitoring/node-exporter-h7mfj" Apr 16 14:00:26.240978 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.240915 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/35436df9-1548-4b04-9e65-ca60b80486e7-node-exporter-wtmp\") pod \"node-exporter-h7mfj\" (UID: \"35436df9-1548-4b04-9e65-ca60b80486e7\") " pod="openshift-monitoring/node-exporter-h7mfj" Apr 16 14:00:26.240978 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.240957 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8qmhl\" (UniqueName: \"kubernetes.io/projected/35436df9-1548-4b04-9e65-ca60b80486e7-kube-api-access-8qmhl\") pod \"node-exporter-h7mfj\" (UID: \"35436df9-1548-4b04-9e65-ca60b80486e7\") " pod="openshift-monitoring/node-exporter-h7mfj" Apr 16 14:00:26.241257 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.240994 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74619757-38ff-48ef-9c0c-129c39811a4d-metrics-client-ca\") pod \"kube-state-metrics-7479c89684-8xh94\" (UID: \"74619757-38ff-48ef-9c0c-129c39811a4d\") " pod="openshift-monitoring/kube-state-metrics-7479c89684-8xh94" Apr 16 14:00:26.241257 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.241028 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/35436df9-1548-4b04-9e65-ca60b80486e7-node-exporter-accelerators-collector-config\") pod \"node-exporter-h7mfj\" (UID: \"35436df9-1548-4b04-9e65-ca60b80486e7\") " pod="openshift-monitoring/node-exporter-h7mfj" Apr 16 14:00:26.241257 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.241059 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7fj2r\" (UniqueName: \"kubernetes.io/projected/74619757-38ff-48ef-9c0c-129c39811a4d-kube-api-access-7fj2r\") pod \"kube-state-metrics-7479c89684-8xh94\" (UID: \"74619757-38ff-48ef-9c0c-129c39811a4d\") " pod="openshift-monitoring/kube-state-metrics-7479c89684-8xh94" Apr 16 14:00:26.241257 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.241064 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/74619757-38ff-48ef-9c0c-129c39811a4d-volume-directive-shadow\") pod \"kube-state-metrics-7479c89684-8xh94\" (UID: \"74619757-38ff-48ef-9c0c-129c39811a4d\") " pod="openshift-monitoring/kube-state-metrics-7479c89684-8xh94" Apr 16 14:00:26.241257 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.241088 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/74619757-38ff-48ef-9c0c-129c39811a4d-kube-state-metrics-tls\") pod \"kube-state-metrics-7479c89684-8xh94\" (UID: \"74619757-38ff-48ef-9c0c-129c39811a4d\") " pod="openshift-monitoring/kube-state-metrics-7479c89684-8xh94" Apr 16 14:00:26.241257 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.241091 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/35436df9-1548-4b04-9e65-ca60b80486e7-node-exporter-wtmp\") pod \"node-exporter-h7mfj\" (UID: \"35436df9-1548-4b04-9e65-ca60b80486e7\") " pod="openshift-monitoring/node-exporter-h7mfj" Apr 16 14:00:26.241257 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.241105 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/35436df9-1548-4b04-9e65-ca60b80486e7-node-exporter-textfile\") pod \"node-exporter-h7mfj\" (UID: \"35436df9-1548-4b04-9e65-ca60b80486e7\") " pod="openshift-monitoring/node-exporter-h7mfj" Apr 16 14:00:26.241257 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.241117 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/35436df9-1548-4b04-9e65-ca60b80486e7-node-exporter-tls\") pod \"node-exporter-h7mfj\" (UID: \"35436df9-1548-4b04-9e65-ca60b80486e7\") " pod="openshift-monitoring/node-exporter-h7mfj" Apr 16 14:00:26.241257 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.241194 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/35436df9-1548-4b04-9e65-ca60b80486e7-sys\") pod \"node-exporter-h7mfj\" (UID: \"35436df9-1548-4b04-9e65-ca60b80486e7\") " pod="openshift-monitoring/node-exporter-h7mfj" Apr 16 14:00:26.241257 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.241235 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/74619757-38ff-48ef-9c0c-129c39811a4d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7479c89684-8xh94\" (UID: \"74619757-38ff-48ef-9c0c-129c39811a4d\") " pod="openshift-monitoring/kube-state-metrics-7479c89684-8xh94" Apr 16 14:00:26.241729 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.241381 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/35436df9-1548-4b04-9e65-ca60b80486e7-sys\") pod \"node-exporter-h7mfj\" (UID: \"35436df9-1548-4b04-9e65-ca60b80486e7\") " pod="openshift-monitoring/node-exporter-h7mfj" Apr 16 14:00:26.241729 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.241666 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74619757-38ff-48ef-9c0c-129c39811a4d-metrics-client-ca\") pod \"kube-state-metrics-7479c89684-8xh94\" (UID: \"74619757-38ff-48ef-9c0c-129c39811a4d\") " pod="openshift-monitoring/kube-state-metrics-7479c89684-8xh94" Apr 16 14:00:26.241860 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.241784 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35436df9-1548-4b04-9e65-ca60b80486e7-metrics-client-ca\") pod \"node-exporter-h7mfj\" (UID: \"35436df9-1548-4b04-9e65-ca60b80486e7\") " pod="openshift-monitoring/node-exporter-h7mfj" Apr 16 14:00:26.241918 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.241845 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/74619757-38ff-48ef-9c0c-129c39811a4d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7479c89684-8xh94\" (UID: \"74619757-38ff-48ef-9c0c-129c39811a4d\") " pod="openshift-monitoring/kube-state-metrics-7479c89684-8xh94" Apr 16 14:00:26.242353 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.242328 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/35436df9-1548-4b04-9e65-ca60b80486e7-node-exporter-accelerators-collector-config\") pod \"node-exporter-h7mfj\" (UID: \"35436df9-1548-4b04-9e65-ca60b80486e7\") " pod="openshift-monitoring/node-exporter-h7mfj" Apr 16 14:00:26.243215 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.243192 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/74619757-38ff-48ef-9c0c-129c39811a4d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7479c89684-8xh94\" (UID: \"74619757-38ff-48ef-9c0c-129c39811a4d\") " pod="openshift-monitoring/kube-state-metrics-7479c89684-8xh94" Apr 16 14:00:26.243447 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.243430 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/74619757-38ff-48ef-9c0c-129c39811a4d-kube-state-metrics-tls\") pod \"kube-state-metrics-7479c89684-8xh94\" (UID: \"74619757-38ff-48ef-9c0c-129c39811a4d\") " pod="openshift-monitoring/kube-state-metrics-7479c89684-8xh94" Apr 16 14:00:26.243832 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.243810 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/35436df9-1548-4b04-9e65-ca60b80486e7-node-exporter-tls\") pod \"node-exporter-h7mfj\" (UID: \"35436df9-1548-4b04-9e65-ca60b80486e7\") " pod="openshift-monitoring/node-exporter-h7mfj" Apr 16 14:00:26.244088 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.244074 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/35436df9-1548-4b04-9e65-ca60b80486e7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h7mfj\" (UID: \"35436df9-1548-4b04-9e65-ca60b80486e7\") " pod="openshift-monitoring/node-exporter-h7mfj" Apr 16 14:00:26.248565 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.248544 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qmhl\" (UniqueName: \"kubernetes.io/projected/35436df9-1548-4b04-9e65-ca60b80486e7-kube-api-access-8qmhl\") pod \"node-exporter-h7mfj\" (UID: \"35436df9-1548-4b04-9e65-ca60b80486e7\") " pod="openshift-monitoring/node-exporter-h7mfj" Apr 16 14:00:26.248811 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.248794 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fj2r\" (UniqueName: \"kubernetes.io/projected/74619757-38ff-48ef-9c0c-129c39811a4d-kube-api-access-7fj2r\") pod \"kube-state-metrics-7479c89684-8xh94\" (UID: \"74619757-38ff-48ef-9c0c-129c39811a4d\") " pod="openshift-monitoring/kube-state-metrics-7479c89684-8xh94" Apr 16 14:00:26.316584 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.316561 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7479c89684-8xh94" Apr 16 14:00:26.321277 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.321251 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-h7mfj" Apr 16 14:00:26.329912 ip-10-0-136-114 kubenswrapper[2574]: W0416 14:00:26.329887 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35436df9_1548_4b04_9e65_ca60b80486e7.slice/crio-62532a4716f27ac771e532996539f10b336c03c4ad1391a6d22cc53210295f1f WatchSource:0}: Error finding container 62532a4716f27ac771e532996539f10b336c03c4ad1391a6d22cc53210295f1f: Status 404 returned error can't find the container with id 62532a4716f27ac771e532996539f10b336c03c4ad1391a6d22cc53210295f1f Apr 16 14:00:26.433673 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.433647 2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7479c89684-8xh94"] Apr 16 14:00:26.436873 ip-10-0-136-114 kubenswrapper[2574]: W0416 14:00:26.436845 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74619757_38ff_48ef_9c0c_129c39811a4d.slice/crio-61fb11abc13a6cf076538128c147add45c93d1e48a03783c7e02d009b41f17e5 WatchSource:0}: Error finding container 61fb11abc13a6cf076538128c147add45c93d1e48a03783c7e02d009b41f17e5: Status 404 returned error can't find the container with id 61fb11abc13a6cf076538128c147add45c93d1e48a03783c7e02d009b41f17e5 Apr 16 14:00:26.644025 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.643992 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d969860-3250-40b5-b124-a01d0c99a8a1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-5669946b84-q57gd\" (UID: \"0d969860-3250-40b5-b124-a01d0c99a8a1\") " pod="openshift-monitoring/openshift-state-metrics-5669946b84-q57gd" Apr 16 14:00:26.646181 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.646160 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d969860-3250-40b5-b124-a01d0c99a8a1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-5669946b84-q57gd\" (UID: \"0d969860-3250-40b5-b124-a01d0c99a8a1\") " pod="openshift-monitoring/openshift-state-metrics-5669946b84-q57gd" Apr 16 14:00:26.895658 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:26.895582 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-5669946b84-q57gd" Apr 16 14:00:27.030891 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.030860 2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-5669946b84-q57gd"] Apr 16 14:00:27.049622 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.049596 2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Apr 16 14:00:27.054308 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.054282 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:00:27.056726 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.056684 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Apr 16 14:00:27.056726 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.056692 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Apr 16 14:00:27.056926 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.056690 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-v2qjl\"" Apr 16 14:00:27.057048 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.057016 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Apr 16 14:00:27.057147 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.057058 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Apr 16 14:00:27.057216 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.057151 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Apr 16 14:00:27.057216 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.057163 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Apr 16 14:00:27.057216 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.057173 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Apr 16 14:00:27.057216 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.057202 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Apr 16 14:00:27.057395 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.057223 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Apr 16 14:00:27.066807 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.066783 2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Apr 16 14:00:27.097064 ip-10-0-136-114 kubenswrapper[2574]: W0416 14:00:27.097034 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d969860_3250_40b5_b124_a01d0c99a8a1.slice/crio-b828f5ce058e26d15028aa2b30ece9ffd8dedc4c34f6a955bd9eb846429a3a8a WatchSource:0}: Error finding container b828f5ce058e26d15028aa2b30ece9ffd8dedc4c34f6a955bd9eb846429a3a8a: Status 404 returned error can't find the container with id b828f5ce058e26d15028aa2b30ece9ffd8dedc4c34f6a955bd9eb846429a3a8a Apr 16 14:00:27.147922 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.147902 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1674f1e9-d0f3-49cf-9c20-40d72e9428e9\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:00:27.147996 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.147942 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-config-out\") pod \"alertmanager-main-0\" (UID: \"1674f1e9-d0f3-49cf-9c20-40d72e9428e9\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:00:27.148053 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.148034 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1674f1e9-d0f3-49cf-9c20-40d72e9428e9\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:00:27.148107 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.148068 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1674f1e9-d0f3-49cf-9c20-40d72e9428e9\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:00:27.148107 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.148097 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-config-volume\") pod \"alertmanager-main-0\" (UID: \"1674f1e9-d0f3-49cf-9c20-40d72e9428e9\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:00:27.148213 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.148121 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1674f1e9-d0f3-49cf-9c20-40d72e9428e9\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:00:27.148213 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.148156 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1674f1e9-d0f3-49cf-9c20-40d72e9428e9\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:00:27.148213 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.148191 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1674f1e9-d0f3-49cf-9c20-40d72e9428e9\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:00:27.148360 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.148224 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1674f1e9-d0f3-49cf-9c20-40d72e9428e9\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:00:27.148360 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.148267 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-web-config\") pod \"alertmanager-main-0\" (UID: \"1674f1e9-d0f3-49cf-9c20-40d72e9428e9\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:00:27.148360 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.148315 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1674f1e9-d0f3-49cf-9c20-40d72e9428e9\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:00:27.148360 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.148354 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1674f1e9-d0f3-49cf-9c20-40d72e9428e9\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:00:27.148535 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.148480 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sbs9\" (UniqueName: \"kubernetes.io/projected/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-kube-api-access-8sbs9\") pod \"alertmanager-main-0\" (UID: \"1674f1e9-d0f3-49cf-9c20-40d72e9428e9\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:00:27.250062 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.250030 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1674f1e9-d0f3-49cf-9c20-40d72e9428e9\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:00:27.250192 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.250083 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1674f1e9-d0f3-49cf-9c20-40d72e9428e9\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:00:27.250192 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.250115 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-config-volume\") pod \"alertmanager-main-0\" (UID: \"1674f1e9-d0f3-49cf-9c20-40d72e9428e9\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:00:27.250192 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.250139 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1674f1e9-d0f3-49cf-9c20-40d72e9428e9\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:00:27.250192 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.250166 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1674f1e9-d0f3-49cf-9c20-40d72e9428e9\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:00:27.250192 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.250191 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1674f1e9-d0f3-49cf-9c20-40d72e9428e9\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:00:27.250471 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.250318 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1674f1e9-d0f3-49cf-9c20-40d72e9428e9\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:00:27.250471 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.250355 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-web-config\") pod \"alertmanager-main-0\" (UID: \"1674f1e9-d0f3-49cf-9c20-40d72e9428e9\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:00:27.250471 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.250396 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1674f1e9-d0f3-49cf-9c20-40d72e9428e9\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:00:27.250471 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.250423 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1674f1e9-d0f3-49cf-9c20-40d72e9428e9\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:00:27.250471 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.250455 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8sbs9\" (UniqueName: \"kubernetes.io/projected/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-kube-api-access-8sbs9\") pod \"alertmanager-main-0\" (UID: \"1674f1e9-d0f3-49cf-9c20-40d72e9428e9\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:00:27.250710 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.250490 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1674f1e9-d0f3-49cf-9c20-40d72e9428e9\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:00:27.250710 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.250520 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-config-out\") pod \"alertmanager-main-0\" (UID: \"1674f1e9-d0f3-49cf-9c20-40d72e9428e9\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:00:27.251494 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.251466 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1674f1e9-d0f3-49cf-9c20-40d72e9428e9\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:00:27.251703 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.251682 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1674f1e9-d0f3-49cf-9c20-40d72e9428e9\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:00:27.252298 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.252028 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1674f1e9-d0f3-49cf-9c20-40d72e9428e9\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:00:27.254212 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.254166 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1674f1e9-d0f3-49cf-9c20-40d72e9428e9\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:00:27.254493 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.254467 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1674f1e9-d0f3-49cf-9c20-40d72e9428e9\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:00:27.255131 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.254944 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-config-volume\") pod \"alertmanager-main-0\" (UID: \"1674f1e9-d0f3-49cf-9c20-40d72e9428e9\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:00:27.255131 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.254980 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1674f1e9-d0f3-49cf-9c20-40d72e9428e9\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:00:27.255131 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.255089 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-web-config\") pod \"alertmanager-main-0\" (UID: \"1674f1e9-d0f3-49cf-9c20-40d72e9428e9\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:00:27.255322 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.255216 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-config-out\") pod \"alertmanager-main-0\" (UID: \"1674f1e9-d0f3-49cf-9c20-40d72e9428e9\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:00:27.255322 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.255304 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1674f1e9-d0f3-49cf-9c20-40d72e9428e9\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:00:27.255423 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.255405 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1674f1e9-d0f3-49cf-9c20-40d72e9428e9\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:00:27.257005 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.256983 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1674f1e9-d0f3-49cf-9c20-40d72e9428e9\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:00:27.258296 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.258274 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sbs9\" (UniqueName: \"kubernetes.io/projected/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-kube-api-access-8sbs9\") pod \"alertmanager-main-0\" (UID: \"1674f1e9-d0f3-49cf-9c20-40d72e9428e9\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:00:27.283011 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.282978 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-5669946b84-q57gd" event={"ID":"0d969860-3250-40b5-b124-a01d0c99a8a1","Type":"ContainerStarted","Data":"082707230d0b0be0d3b2d35b735b051a978e4e6081167645bfc79f573c2f94e9"} Apr 16 14:00:27.283121 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.283022 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-5669946b84-q57gd" event={"ID":"0d969860-3250-40b5-b124-a01d0c99a8a1","Type":"ContainerStarted","Data":"b6f77fa4bfaa45a376815d4586e2728eea668bfc8b6c8e1aab15366801d19ea1"} Apr 16 14:00:27.283121 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.283038 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-5669946b84-q57gd" event={"ID":"0d969860-3250-40b5-b124-a01d0c99a8a1","Type":"ContainerStarted","Data":"b828f5ce058e26d15028aa2b30ece9ffd8dedc4c34f6a955bd9eb846429a3a8a"} Apr 16 14:00:27.284009 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.283974 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7479c89684-8xh94" event={"ID":"74619757-38ff-48ef-9c0c-129c39811a4d","Type":"ContainerStarted","Data":"61fb11abc13a6cf076538128c147add45c93d1e48a03783c7e02d009b41f17e5"} Apr 16 14:00:27.285316 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.285291 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h7mfj" event={"ID":"35436df9-1548-4b04-9e65-ca60b80486e7","Type":"ContainerStarted","Data":"9ba80486a53aae5e22161303dcc9d5ed9246592b82cc0edb243be37c3264d45b"} Apr 16 14:00:27.285397 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.285328 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h7mfj" event={"ID":"35436df9-1548-4b04-9e65-ca60b80486e7","Type":"ContainerStarted","Data":"62532a4716f27ac771e532996539f10b336c03c4ad1391a6d22cc53210295f1f"} Apr 16 14:00:27.366772 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.366734 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:00:27.488927 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.488900 2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Apr 16 14:00:27.491498 ip-10-0-136-114 kubenswrapper[2574]: W0416 14:00:27.491463 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1674f1e9_d0f3_49cf_9c20_40d72e9428e9.slice/crio-3e07d94cd968c488d5f1c5c3bf270784acc007d3e3e0c57e51133a06d7b0b0c4 WatchSource:0}: Error finding container 3e07d94cd968c488d5f1c5c3bf270784acc007d3e3e0c57e51133a06d7b0b0c4: Status 404 returned error can't find the container with id 3e07d94cd968c488d5f1c5c3bf270784acc007d3e3e0c57e51133a06d7b0b0c4 Apr 16 14:00:27.983932 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.983903 2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7c94669ccd-k5vvr"] Apr 16 14:00:27.988252 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.988230 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7c94669ccd-k5vvr" Apr 16 14:00:27.990424 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.990402 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Apr 16 14:00:27.991783 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.991733 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Apr 16 14:00:27.992066 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.991993 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Apr 16 14:00:27.992193 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.992129 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Apr 16 14:00:27.992269 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.992245 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-51cdjqbjd1ho9\"" Apr 16 14:00:27.992757 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.992727 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Apr 16 14:00:27.994141 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:27.994095 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-4pgtg\"" Apr 16 14:00:28.006641 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:28.006624 2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7c94669ccd-k5vvr"] Apr 16 14:00:28.057285 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:28.057249 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/78e0f4b8-9966-439a-98af-e8d7a5253a42-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7c94669ccd-k5vvr\" (UID: \"78e0f4b8-9966-439a-98af-e8d7a5253a42\") " pod="openshift-monitoring/thanos-querier-7c94669ccd-k5vvr" Apr 16 14:00:28.057648 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:28.057329 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/78e0f4b8-9966-439a-98af-e8d7a5253a42-metrics-client-ca\") pod \"thanos-querier-7c94669ccd-k5vvr\" (UID: \"78e0f4b8-9966-439a-98af-e8d7a5253a42\") " pod="openshift-monitoring/thanos-querier-7c94669ccd-k5vvr" Apr 16 14:00:28.057648 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:28.057429 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/78e0f4b8-9966-439a-98af-e8d7a5253a42-secret-grpc-tls\") pod \"thanos-querier-7c94669ccd-k5vvr\" (UID: \"78e0f4b8-9966-439a-98af-e8d7a5253a42\") " pod="openshift-monitoring/thanos-querier-7c94669ccd-k5vvr" Apr 16 14:00:28.057648 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:28.057460 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmdx5\" (UniqueName: \"kubernetes.io/projected/78e0f4b8-9966-439a-98af-e8d7a5253a42-kube-api-access-jmdx5\") pod \"thanos-querier-7c94669ccd-k5vvr\" (UID: \"78e0f4b8-9966-439a-98af-e8d7a5253a42\") " pod="openshift-monitoring/thanos-querier-7c94669ccd-k5vvr" Apr 16 14:00:28.057648 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:28.057498 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/78e0f4b8-9966-439a-98af-e8d7a5253a42-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7c94669ccd-k5vvr\" (UID: \"78e0f4b8-9966-439a-98af-e8d7a5253a42\") " pod="openshift-monitoring/thanos-querier-7c94669ccd-k5vvr" Apr 16 14:00:28.057648 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:28.057555 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/78e0f4b8-9966-439a-98af-e8d7a5253a42-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7c94669ccd-k5vvr\" (UID: \"78e0f4b8-9966-439a-98af-e8d7a5253a42\") " pod="openshift-monitoring/thanos-querier-7c94669ccd-k5vvr" Apr 16 14:00:28.057891 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:28.057649 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/78e0f4b8-9966-439a-98af-e8d7a5253a42-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7c94669ccd-k5vvr\" (UID: \"78e0f4b8-9966-439a-98af-e8d7a5253a42\") " pod="openshift-monitoring/thanos-querier-7c94669ccd-k5vvr" Apr 16 14:00:28.057891 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:28.057695 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/78e0f4b8-9966-439a-98af-e8d7a5253a42-secret-thanos-querier-tls\") pod \"thanos-querier-7c94669ccd-k5vvr\" (UID: \"78e0f4b8-9966-439a-98af-e8d7a5253a42\") " pod="openshift-monitoring/thanos-querier-7c94669ccd-k5vvr" Apr 16 14:00:28.174629 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:28.174597 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/78e0f4b8-9966-439a-98af-e8d7a5253a42-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7c94669ccd-k5vvr\" (UID: \"78e0f4b8-9966-439a-98af-e8d7a5253a42\") " pod="openshift-monitoring/thanos-querier-7c94669ccd-k5vvr" Apr 16 14:00:28.174767 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:28.174660 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/78e0f4b8-9966-439a-98af-e8d7a5253a42-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7c94669ccd-k5vvr\" (UID: \"78e0f4b8-9966-439a-98af-e8d7a5253a42\") " pod="openshift-monitoring/thanos-querier-7c94669ccd-k5vvr" Apr 16 14:00:28.174767 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:28.174692 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/78e0f4b8-9966-439a-98af-e8d7a5253a42-secret-thanos-querier-tls\") pod \"thanos-querier-7c94669ccd-k5vvr\" (UID: \"78e0f4b8-9966-439a-98af-e8d7a5253a42\") " pod="openshift-monitoring/thanos-querier-7c94669ccd-k5vvr" Apr 16 14:00:28.174767 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:28.174733 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/78e0f4b8-9966-439a-98af-e8d7a5253a42-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7c94669ccd-k5vvr\" (UID: \"78e0f4b8-9966-439a-98af-e8d7a5253a42\") " pod="openshift-monitoring/thanos-querier-7c94669ccd-k5vvr" Apr 16 14:00:28.174928 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:28.174854 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/78e0f4b8-9966-439a-98af-e8d7a5253a42-metrics-client-ca\") pod \"thanos-querier-7c94669ccd-k5vvr\" (UID: \"78e0f4b8-9966-439a-98af-e8d7a5253a42\") " pod="openshift-monitoring/thanos-querier-7c94669ccd-k5vvr" Apr 16 14:00:28.174980 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:28.174927 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/78e0f4b8-9966-439a-98af-e8d7a5253a42-secret-grpc-tls\") pod \"thanos-querier-7c94669ccd-k5vvr\" (UID: \"78e0f4b8-9966-439a-98af-e8d7a5253a42\") " pod="openshift-monitoring/thanos-querier-7c94669ccd-k5vvr" Apr 16 14:00:28.174980 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:28.174957 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jmdx5\" (UniqueName: \"kubernetes.io/projected/78e0f4b8-9966-439a-98af-e8d7a5253a42-kube-api-access-jmdx5\") pod \"thanos-querier-7c94669ccd-k5vvr\" (UID: \"78e0f4b8-9966-439a-98af-e8d7a5253a42\") " pod="openshift-monitoring/thanos-querier-7c94669ccd-k5vvr" Apr 16 14:00:28.175076 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:28.175011 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/78e0f4b8-9966-439a-98af-e8d7a5253a42-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7c94669ccd-k5vvr\" (UID: \"78e0f4b8-9966-439a-98af-e8d7a5253a42\") " pod="openshift-monitoring/thanos-querier-7c94669ccd-k5vvr" Apr 16 14:00:28.175594 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:28.175564 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/78e0f4b8-9966-439a-98af-e8d7a5253a42-metrics-client-ca\") pod \"thanos-querier-7c94669ccd-k5vvr\" (UID: \"78e0f4b8-9966-439a-98af-e8d7a5253a42\") " pod="openshift-monitoring/thanos-querier-7c94669ccd-k5vvr" Apr 16 14:00:28.177995 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:28.177963 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/78e0f4b8-9966-439a-98af-e8d7a5253a42-secret-thanos-querier-tls\") pod \"thanos-querier-7c94669ccd-k5vvr\" (UID: \"78e0f4b8-9966-439a-98af-e8d7a5253a42\") " pod="openshift-monitoring/thanos-querier-7c94669ccd-k5vvr" Apr 16 14:00:28.178128 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:28.178092 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/78e0f4b8-9966-439a-98af-e8d7a5253a42-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7c94669ccd-k5vvr\" (UID: \"78e0f4b8-9966-439a-98af-e8d7a5253a42\") " pod="openshift-monitoring/thanos-querier-7c94669ccd-k5vvr" Apr 16 14:00:28.178238 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:28.178178 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/78e0f4b8-9966-439a-98af-e8d7a5253a42-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7c94669ccd-k5vvr\" (UID: \"78e0f4b8-9966-439a-98af-e8d7a5253a42\") " pod="openshift-monitoring/thanos-querier-7c94669ccd-k5vvr" Apr 16 14:00:28.178623 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:28.178595 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/78e0f4b8-9966-439a-98af-e8d7a5253a42-secret-grpc-tls\") pod \"thanos-querier-7c94669ccd-k5vvr\" (UID: \"78e0f4b8-9966-439a-98af-e8d7a5253a42\") " pod="openshift-monitoring/thanos-querier-7c94669ccd-k5vvr" Apr 16 14:00:28.178805 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:28.178781 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/78e0f4b8-9966-439a-98af-e8d7a5253a42-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7c94669ccd-k5vvr\" (UID: \"78e0f4b8-9966-439a-98af-e8d7a5253a42\") " pod="openshift-monitoring/thanos-querier-7c94669ccd-k5vvr" Apr 16 14:00:28.179681 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:28.179661 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/78e0f4b8-9966-439a-98af-e8d7a5253a42-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7c94669ccd-k5vvr\" (UID: \"78e0f4b8-9966-439a-98af-e8d7a5253a42\") " pod="openshift-monitoring/thanos-querier-7c94669ccd-k5vvr" Apr 16 14:00:28.184210 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:28.184187 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmdx5\" (UniqueName: \"kubernetes.io/projected/78e0f4b8-9966-439a-98af-e8d7a5253a42-kube-api-access-jmdx5\") pod \"thanos-querier-7c94669ccd-k5vvr\" (UID: \"78e0f4b8-9966-439a-98af-e8d7a5253a42\") " pod="openshift-monitoring/thanos-querier-7c94669ccd-k5vvr" Apr 16 14:00:28.289684 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:28.289653 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7479c89684-8xh94" event={"ID":"74619757-38ff-48ef-9c0c-129c39811a4d","Type":"ContainerStarted","Data":"18ccbbdbf705eebb3729a626204aeddd77c7f2d683edcce7e79c7f963491cc99"} Apr 16 14:00:28.291020 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:28.290975 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1674f1e9-d0f3-49cf-9c20-40d72e9428e9","Type":"ContainerStarted","Data":"3e07d94cd968c488d5f1c5c3bf270784acc007d3e3e0c57e51133a06d7b0b0c4"} Apr 16 14:00:28.293008 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:28.292903 2574 generic.go:358] "Generic (PLEG): container finished" podID="35436df9-1548-4b04-9e65-ca60b80486e7" containerID="9ba80486a53aae5e22161303dcc9d5ed9246592b82cc0edb243be37c3264d45b" exitCode=0 Apr 16 14:00:28.293008 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:28.292956 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h7mfj" event={"ID":"35436df9-1548-4b04-9e65-ca60b80486e7","Type":"ContainerDied","Data":"9ba80486a53aae5e22161303dcc9d5ed9246592b82cc0edb243be37c3264d45b"} Apr 16 14:00:28.299329 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:28.299301 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7c94669ccd-k5vvr" Apr 16 14:00:28.478584 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:28.478473 2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7c94669ccd-k5vvr"] Apr 16 14:00:28.481503 ip-10-0-136-114 kubenswrapper[2574]: W0416 14:00:28.481476 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78e0f4b8_9966_439a_98af_e8d7a5253a42.slice/crio-ea8d211455459e0e55207cf1a62cfd7ffe71e7f3a0912df8f704d5fc61041d7f WatchSource:0}: Error finding container ea8d211455459e0e55207cf1a62cfd7ffe71e7f3a0912df8f704d5fc61041d7f: Status 404 returned error can't find the container with id ea8d211455459e0e55207cf1a62cfd7ffe71e7f3a0912df8f704d5fc61041d7f Apr 16 14:00:29.297653 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:29.297564 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c94669ccd-k5vvr" event={"ID":"78e0f4b8-9966-439a-98af-e8d7a5253a42","Type":"ContainerStarted","Data":"ea8d211455459e0e55207cf1a62cfd7ffe71e7f3a0912df8f704d5fc61041d7f"} Apr 16 14:00:29.299725 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:29.299697 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h7mfj" event={"ID":"35436df9-1548-4b04-9e65-ca60b80486e7","Type":"ContainerStarted","Data":"389b2f81ce1980aadf2fa35999fa138c68d20c6aeaf45736c535922df008de30"} Apr 16 14:00:29.299863 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:29.299729 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h7mfj" event={"ID":"35436df9-1548-4b04-9e65-ca60b80486e7","Type":"ContainerStarted","Data":"f7edeaa6df1e115fe10b952e141388a34b8c047c645844c452fb8f43625574e0"} Apr 16 14:00:29.301590 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:29.301537 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7479c89684-8xh94" event={"ID":"74619757-38ff-48ef-9c0c-129c39811a4d","Type":"ContainerStarted","Data":"d23a92157ad078773b5ae6adb9b6ce4c42a834548c727ad88b83ed1ccd9a445d"} Apr 16 14:00:29.301590 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:29.301570 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7479c89684-8xh94" event={"ID":"74619757-38ff-48ef-9c0c-129c39811a4d","Type":"ContainerStarted","Data":"492fc52fe0dfe229c3a19e7e7fe8c259410b9a479dfa64d3a4b214670b65a5c6"} Apr 16 14:00:29.302997 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:29.302958 2574 generic.go:358] "Generic (PLEG): container finished" podID="1674f1e9-d0f3-49cf-9c20-40d72e9428e9" containerID="fc8488bc4c0ac26656b5829766f499e968a418eda525d0881b36c8ae0d3881de" exitCode=0 Apr 16 14:00:29.303079 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:29.303005 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1674f1e9-d0f3-49cf-9c20-40d72e9428e9","Type":"ContainerDied","Data":"fc8488bc4c0ac26656b5829766f499e968a418eda525d0881b36c8ae0d3881de"} Apr 16 14:00:29.324760 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:29.324551 2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-h7mfj" podStartSLOduration=3.5088951379999997 podStartE2EDuration="4.324535518s" podCreationTimestamp="2026-04-16 14:00:25 +0000 UTC" firstStartedPulling="2026-04-16 14:00:26.331602847 +0000 UTC m=+196.161337695" lastFinishedPulling="2026-04-16 14:00:27.147243213 +0000 UTC m=+196.976978075" observedRunningTime="2026-04-16 14:00:29.323574424 +0000 UTC m=+199.153309296" watchObservedRunningTime="2026-04-16 14:00:29.324535518 +0000 UTC m=+199.154270402" Apr 16 14:00:29.342249 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:29.342205 2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7479c89684-8xh94" podStartSLOduration=2.599622413 podStartE2EDuration="4.342194634s" podCreationTimestamp="2026-04-16 14:00:25 +0000 UTC" firstStartedPulling="2026-04-16 14:00:26.438631928 +0000 UTC m=+196.268366775" lastFinishedPulling="2026-04-16 14:00:28.181204147 +0000 UTC m=+198.010938996" observedRunningTime="2026-04-16 14:00:29.340886494 +0000 UTC m=+199.170621367" watchObservedRunningTime="2026-04-16 14:00:29.342194634 +0000 UTC m=+199.171929503" Apr 16 14:00:31.311626 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:31.311586 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1674f1e9-d0f3-49cf-9c20-40d72e9428e9","Type":"ContainerStarted","Data":"8a107c531e665503ab11c11b3c3393f93b4baf07bed584f9d77e9b0b398d2400"} Apr 16 14:00:31.312061 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:31.311637 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1674f1e9-d0f3-49cf-9c20-40d72e9428e9","Type":"ContainerStarted","Data":"9c1100377039bff38551f96b0a7bb53031da5e82e496d8d6f7e56e48d24a9c89"} Apr 16 14:00:31.312061 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:31.311649 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1674f1e9-d0f3-49cf-9c20-40d72e9428e9","Type":"ContainerStarted","Data":"2a98591981bae2c95137ab5cb7fa07f6e15c6fd690e1d01717204617f5248513"} Apr 16 14:00:31.312061 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:31.311662 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1674f1e9-d0f3-49cf-9c20-40d72e9428e9","Type":"ContainerStarted","Data":"c98b6e56106018578f36bcfd1e52ce2f0a937c05220ba15225e72d68577b5675"} Apr 16 14:00:31.313997 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:31.313952 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c94669ccd-k5vvr" event={"ID":"78e0f4b8-9966-439a-98af-e8d7a5253a42","Type":"ContainerStarted","Data":"50020f43049857bfad55fffd04797ec6afb86d0869ced5e654ce2a057ed6f8b9"} Apr 16 14:00:31.313997 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:31.313991 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c94669ccd-k5vvr" event={"ID":"78e0f4b8-9966-439a-98af-e8d7a5253a42","Type":"ContainerStarted","Data":"b5c0a112b606b79d5d89b7b6b4ebe20e20d6372f7ca957282c0d58cd96d2af40"} Apr 16 14:00:31.314176 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:31.314004 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c94669ccd-k5vvr" event={"ID":"78e0f4b8-9966-439a-98af-e8d7a5253a42","Type":"ContainerStarted","Data":"1769b394bc08f98acba84aaa518f5b7e6cbe42e32133bff60981b074c6f0de0c"} Apr 16 14:00:31.315981 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:31.315954 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-5669946b84-q57gd" event={"ID":"0d969860-3250-40b5-b124-a01d0c99a8a1","Type":"ContainerStarted","Data":"9a8f77f3d9e2f5c367e76dfb9a29c4383f7dafa519df743e39a9870ecfbbbb6f"} Apr 16 14:00:31.338282 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:31.338228 2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-5669946b84-q57gd" podStartSLOduration=2.5681398250000003 podStartE2EDuration="6.338213173s" podCreationTimestamp="2026-04-16 14:00:25 +0000 UTC" firstStartedPulling="2026-04-16 14:00:27.229939951 +0000 UTC m=+197.059674802" lastFinishedPulling="2026-04-16 14:00:31.000013277 +0000 UTC m=+200.829748150" observedRunningTime="2026-04-16 14:00:31.336960553 +0000 UTC m=+201.166695424" watchObservedRunningTime="2026-04-16 14:00:31.338213173 +0000 UTC m=+201.167948045" Apr 16 14:00:32.321338 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:32.321302 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1674f1e9-d0f3-49cf-9c20-40d72e9428e9","Type":"ContainerStarted","Data":"57ef3cdb8657c272c6327739170ad1fc5a64a07e80709e961b8c458ec9cbbe51"} Apr 16 14:00:33.326431 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:33.326357 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c94669ccd-k5vvr" event={"ID":"78e0f4b8-9966-439a-98af-e8d7a5253a42","Type":"ContainerStarted","Data":"9815a33f4e6247a82e98ed7dadc3cffb2e1d56ff6cb7b0cc08b7ad3d3ff9f79f"} Apr 16 14:00:33.326431 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:33.326392 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c94669ccd-k5vvr" event={"ID":"78e0f4b8-9966-439a-98af-e8d7a5253a42","Type":"ContainerStarted","Data":"f633d991e93466f33ec3c9461882491ff631560624e675050868438ac348d38d"} Apr 16 14:00:33.326431 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:33.326402 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c94669ccd-k5vvr" event={"ID":"78e0f4b8-9966-439a-98af-e8d7a5253a42","Type":"ContainerStarted","Data":"315a6e767ecaa2d84d66dd320912a77264b99ee6051628f497a5bbc97980c9dd"} Apr 16 14:00:33.326855 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:33.326500 2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7c94669ccd-k5vvr" Apr 16 14:00:33.349276 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:33.349226 2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7c94669ccd-k5vvr" podStartSLOduration=1.982085045 podStartE2EDuration="6.349212654s" podCreationTimestamp="2026-04-16 14:00:27 +0000 UTC" firstStartedPulling="2026-04-16 14:00:28.484051585 +0000 UTC m=+198.313786433" lastFinishedPulling="2026-04-16 14:00:32.851179192 +0000 UTC m=+202.680914042" observedRunningTime="2026-04-16 14:00:33.347102369 +0000 UTC m=+203.176837238" watchObservedRunningTime="2026-04-16 14:00:33.349212654 +0000 UTC m=+203.178947540" Apr 16 14:00:34.331504 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:34.331464 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1674f1e9-d0f3-49cf-9c20-40d72e9428e9","Type":"ContainerStarted","Data":"cc125dfb019a0fc9f2aa8b910005d58e61f61ca47f96df371db6bd86f9c08d0f"} Apr 16 14:00:34.379571 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:34.379518 2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=1.306041972 podStartE2EDuration="7.379503442s" podCreationTimestamp="2026-04-16 14:00:27 +0000 UTC" firstStartedPulling="2026-04-16 14:00:27.49340602 +0000 UTC m=+197.323140869" lastFinishedPulling="2026-04-16 14:00:33.566867487 +0000 UTC m=+203.396602339" observedRunningTime="2026-04-16 14:00:34.378579445 +0000 UTC m=+204.208314315" watchObservedRunningTime="2026-04-16 14:00:34.379503442 +0000 UTC m=+204.209238311" Apr 16 14:00:36.344794 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:36.344727 2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-image-registry/image-registry-b76cd7dd-grjcr" podUID="eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac" containerName="registry" containerID="cri-o://3324f8d8af0df86b61f76ef9abb9f8496e75681cb3fd39053fe38fd20bf5271f" gracePeriod=30 Apr 16 14:00:36.577587 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:36.577565 2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b76cd7dd-grjcr" Apr 16 14:00:36.750856 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:36.750788 2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac-image-registry-private-configuration\") pod \"eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac\" (UID: \"eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac\") " Apr 16 14:00:36.750856 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:36.750822 2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac-bound-sa-token\") pod \"eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac\" (UID: \"eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac\") " Apr 16 14:00:36.750856 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:36.750839 2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac-registry-tls\") pod \"eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac\" (UID: \"eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac\") " Apr 16 14:00:36.751104 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:36.750875 2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pdcsf\" (UniqueName: \"kubernetes.io/projected/eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac-kube-api-access-pdcsf\") pod \"eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac\" (UID: \"eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac\") " Apr 16 14:00:36.751104 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:36.750907 2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac-registry-certificates\") pod \"eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac\" (UID: \"eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac\") " Apr 16 14:00:36.751104 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:36.750934 2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac-installation-pull-secrets\") pod \"eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac\" (UID: \"eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac\") " Apr 16 14:00:36.751104 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:36.750962 2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac-ca-trust-extracted\") pod \"eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac\" (UID: \"eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac\") " Apr 16 14:00:36.751104 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:36.751007 2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac-trusted-ca\") pod \"eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac\" (UID: \"eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac\") " Apr 16 14:00:36.751503 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:36.751427 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac" (UID: "eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 16 14:00:36.751622 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:36.751528 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac" (UID: "eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 16 14:00:36.753307 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:36.753271 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac" (UID: "eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 16 14:00:36.753518 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:36.753395 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac" (UID: "eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 16 14:00:36.753518 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:36.753487 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac-kube-api-access-pdcsf" (OuterVolumeSpecName: "kube-api-access-pdcsf") pod "eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac" (UID: "eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac"). InnerVolumeSpecName "kube-api-access-pdcsf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 16 14:00:36.753648 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:36.753523 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac" (UID: "eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 16 14:00:36.753648 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:36.753626 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac" (UID: "eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 16 14:00:36.759955 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:36.759932 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac" (UID: "eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 16 14:00:36.851549 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:36.851530 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdcsf\" (UniqueName: \"kubernetes.io/projected/eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac-kube-api-access-pdcsf\") on node \"ip-10-0-136-114.ec2.internal\" DevicePath \"\"" Apr 16 14:00:36.851549 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:36.851550 2574 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac-registry-certificates\") on node \"ip-10-0-136-114.ec2.internal\" DevicePath \"\"" Apr 16 14:00:36.851669 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:36.851561 2574 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac-installation-pull-secrets\") on node \"ip-10-0-136-114.ec2.internal\" DevicePath \"\"" Apr 16 14:00:36.851669 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:36.851570 2574 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac-ca-trust-extracted\") on node \"ip-10-0-136-114.ec2.internal\" DevicePath \"\"" Apr 16 14:00:36.851669 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:36.851592 2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac-trusted-ca\") on node \"ip-10-0-136-114.ec2.internal\" DevicePath \"\"" Apr 16 14:00:36.851669 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:36.851601 2574 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac-image-registry-private-configuration\") on node \"ip-10-0-136-114.ec2.internal\" DevicePath \"\"" Apr 16 14:00:36.851669 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:36.851611 2574 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac-bound-sa-token\") on node \"ip-10-0-136-114.ec2.internal\" DevicePath \"\"" Apr 16 14:00:36.851669 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:36.851620 2574 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac-registry-tls\") on node \"ip-10-0-136-114.ec2.internal\" DevicePath \"\"" Apr 16 14:00:37.339378 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:37.339342 2574 generic.go:358] "Generic (PLEG): container finished" podID="eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac" containerID="3324f8d8af0df86b61f76ef9abb9f8496e75681cb3fd39053fe38fd20bf5271f" exitCode=0 Apr 16 14:00:37.339569 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:37.339409 2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b76cd7dd-grjcr" Apr 16 14:00:37.339569 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:37.339412 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b76cd7dd-grjcr" event={"ID":"eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac","Type":"ContainerDied","Data":"3324f8d8af0df86b61f76ef9abb9f8496e75681cb3fd39053fe38fd20bf5271f"} Apr 16 14:00:37.339569 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:37.339516 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b76cd7dd-grjcr" event={"ID":"eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac","Type":"ContainerDied","Data":"8c0ee2c300c34de93230429ffafcc2f3220b46ecbb2108dc0209423829ab7913"} Apr 16 14:00:37.339569 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:37.339531 2574 scope.go:117] "RemoveContainer" containerID="3324f8d8af0df86b61f76ef9abb9f8496e75681cb3fd39053fe38fd20bf5271f" Apr 16 14:00:37.347339 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:37.347293 2574 scope.go:117] "RemoveContainer" containerID="3324f8d8af0df86b61f76ef9abb9f8496e75681cb3fd39053fe38fd20bf5271f" Apr 16 14:00:37.347609 ip-10-0-136-114 kubenswrapper[2574]: E0416 14:00:37.347581 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3324f8d8af0df86b61f76ef9abb9f8496e75681cb3fd39053fe38fd20bf5271f\": container with ID starting with 3324f8d8af0df86b61f76ef9abb9f8496e75681cb3fd39053fe38fd20bf5271f not found: ID does not exist" containerID="3324f8d8af0df86b61f76ef9abb9f8496e75681cb3fd39053fe38fd20bf5271f" Apr 16 14:00:37.347648 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:37.347610 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3324f8d8af0df86b61f76ef9abb9f8496e75681cb3fd39053fe38fd20bf5271f"} err="failed to get container status \"3324f8d8af0df86b61f76ef9abb9f8496e75681cb3fd39053fe38fd20bf5271f\": rpc error: code = NotFound desc = could not find container \"3324f8d8af0df86b61f76ef9abb9f8496e75681cb3fd39053fe38fd20bf5271f\": container with ID starting with 3324f8d8af0df86b61f76ef9abb9f8496e75681cb3fd39053fe38fd20bf5271f not found: ID does not exist" Apr 16 14:00:37.355227 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:37.355207 2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-b76cd7dd-grjcr"] Apr 16 14:00:37.363464 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:37.363442 2574 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-b76cd7dd-grjcr"] Apr 16 14:00:38.811951 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:38.810250 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac" path="/var/lib/kubelet/pods/eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac/volumes" Apr 16 14:00:39.336908 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:39.336881 2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7c94669ccd-k5vvr" Apr 16 14:00:39.973731 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:39.973695 2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-586b57c7b4-bd2j9"] Apr 16 14:00:39.974172 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:39.974152 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac" containerName="registry" Apr 16 14:00:39.974241 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:39.974176 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac" containerName="registry" Apr 16 14:00:39.974292 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:39.974251 2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="eb098e1c-6a7b-4d8b-a47e-91a6a1a2f5ac" containerName="registry" Apr 16 14:00:39.979275 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:39.979253 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-586b57c7b4-bd2j9" Apr 16 14:00:39.981646 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:39.981622 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Apr 16 14:00:39.981764 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:39.981664 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Apr 16 14:00:39.981764 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:39.981664 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-8zxrl\"" Apr 16 14:00:39.987793 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:39.987773 2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-586b57c7b4-bd2j9"] Apr 16 14:00:40.080435 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:40.080411 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pgh5\" (UniqueName: \"kubernetes.io/projected/9b1f9023-4bfb-4ee6-9045-780848f02427-kube-api-access-6pgh5\") pod \"downloads-586b57c7b4-bd2j9\" (UID: \"9b1f9023-4bfb-4ee6-9045-780848f02427\") " pod="openshift-console/downloads-586b57c7b4-bd2j9" Apr 16 14:00:40.181175 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:40.181143 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6pgh5\" (UniqueName: \"kubernetes.io/projected/9b1f9023-4bfb-4ee6-9045-780848f02427-kube-api-access-6pgh5\") pod \"downloads-586b57c7b4-bd2j9\" (UID: \"9b1f9023-4bfb-4ee6-9045-780848f02427\") " pod="openshift-console/downloads-586b57c7b4-bd2j9" Apr 16 14:00:40.189200 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:40.189176 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pgh5\" (UniqueName: \"kubernetes.io/projected/9b1f9023-4bfb-4ee6-9045-780848f02427-kube-api-access-6pgh5\") pod \"downloads-586b57c7b4-bd2j9\" (UID: \"9b1f9023-4bfb-4ee6-9045-780848f02427\") " pod="openshift-console/downloads-586b57c7b4-bd2j9" Apr 16 14:00:40.289094 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:40.289035 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-586b57c7b4-bd2j9" Apr 16 14:00:40.404680 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:40.404533 2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-586b57c7b4-bd2j9"] Apr 16 14:00:40.407228 ip-10-0-136-114 kubenswrapper[2574]: W0416 14:00:40.407202 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b1f9023_4bfb_4ee6_9045_780848f02427.slice/crio-b1eedb173d6e1ae8dd867c6ab2c365e235af4df350d95ef565c9f4978d638ebb WatchSource:0}: Error finding container b1eedb173d6e1ae8dd867c6ab2c365e235af4df350d95ef565c9f4978d638ebb: Status 404 returned error can't find the container with id b1eedb173d6e1ae8dd867c6ab2c365e235af4df350d95ef565c9f4978d638ebb Apr 16 14:00:41.352496 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:41.352457 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-586b57c7b4-bd2j9" event={"ID":"9b1f9023-4bfb-4ee6-9045-780848f02427","Type":"ContainerStarted","Data":"b1eedb173d6e1ae8dd867c6ab2c365e235af4df350d95ef565c9f4978d638ebb"} Apr 16 14:00:44.089600 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:44.089559 2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-765cb78b68-dqp8h"] Apr 16 14:00:44.093158 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:44.093132 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-765cb78b68-dqp8h" Apr 16 14:00:44.096217 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:44.096191 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Apr 16 14:00:44.096217 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:44.096206 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Apr 16 14:00:44.096396 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:44.096267 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Apr 16 14:00:44.096396 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:44.096206 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Apr 16 14:00:44.096396 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:44.096331 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-szv54\"" Apr 16 14:00:44.096612 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:44.096559 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Apr 16 14:00:44.103437 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:44.103416 2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-765cb78b68-dqp8h"] Apr 16 14:00:44.217208 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:44.217171 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4npnf\" (UniqueName: \"kubernetes.io/projected/ee5dbe45-5126-4c3e-9397-69e2153637de-kube-api-access-4npnf\") pod \"console-765cb78b68-dqp8h\" (UID: \"ee5dbe45-5126-4c3e-9397-69e2153637de\") " pod="openshift-console/console-765cb78b68-dqp8h" Apr 16 14:00:44.217356 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:44.217316 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ee5dbe45-5126-4c3e-9397-69e2153637de-oauth-serving-cert\") pod \"console-765cb78b68-dqp8h\" (UID: \"ee5dbe45-5126-4c3e-9397-69e2153637de\") " pod="openshift-console/console-765cb78b68-dqp8h" Apr 16 14:00:44.217411 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:44.217364 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ee5dbe45-5126-4c3e-9397-69e2153637de-console-serving-cert\") pod \"console-765cb78b68-dqp8h\" (UID: \"ee5dbe45-5126-4c3e-9397-69e2153637de\") " pod="openshift-console/console-765cb78b68-dqp8h" Apr 16 14:00:44.217411 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:44.217402 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ee5dbe45-5126-4c3e-9397-69e2153637de-console-oauth-config\") pod \"console-765cb78b68-dqp8h\" (UID: \"ee5dbe45-5126-4c3e-9397-69e2153637de\") " pod="openshift-console/console-765cb78b68-dqp8h" Apr 16 14:00:44.217516 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:44.217437 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ee5dbe45-5126-4c3e-9397-69e2153637de-service-ca\") pod \"console-765cb78b68-dqp8h\" (UID: \"ee5dbe45-5126-4c3e-9397-69e2153637de\") " pod="openshift-console/console-765cb78b68-dqp8h" Apr 16 14:00:44.217516 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:44.217484 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ee5dbe45-5126-4c3e-9397-69e2153637de-console-config\") pod \"console-765cb78b68-dqp8h\" (UID: \"ee5dbe45-5126-4c3e-9397-69e2153637de\") " pod="openshift-console/console-765cb78b68-dqp8h" Apr 16 14:00:44.318397 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:44.318361 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ee5dbe45-5126-4c3e-9397-69e2153637de-oauth-serving-cert\") pod \"console-765cb78b68-dqp8h\" (UID: \"ee5dbe45-5126-4c3e-9397-69e2153637de\") " pod="openshift-console/console-765cb78b68-dqp8h" Apr 16 14:00:44.318550 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:44.318419 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ee5dbe45-5126-4c3e-9397-69e2153637de-console-serving-cert\") pod \"console-765cb78b68-dqp8h\" (UID: \"ee5dbe45-5126-4c3e-9397-69e2153637de\") " pod="openshift-console/console-765cb78b68-dqp8h" Apr 16 14:00:44.318550 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:44.318457 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ee5dbe45-5126-4c3e-9397-69e2153637de-console-oauth-config\") pod \"console-765cb78b68-dqp8h\" (UID: \"ee5dbe45-5126-4c3e-9397-69e2153637de\") " pod="openshift-console/console-765cb78b68-dqp8h" Apr 16 14:00:44.318550 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:44.318490 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ee5dbe45-5126-4c3e-9397-69e2153637de-service-ca\") pod \"console-765cb78b68-dqp8h\" (UID: \"ee5dbe45-5126-4c3e-9397-69e2153637de\") " pod="openshift-console/console-765cb78b68-dqp8h" Apr 16 14:00:44.318550 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:44.318536 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ee5dbe45-5126-4c3e-9397-69e2153637de-console-config\") pod \"console-765cb78b68-dqp8h\" (UID: \"ee5dbe45-5126-4c3e-9397-69e2153637de\") " pod="openshift-console/console-765cb78b68-dqp8h" Apr 16 14:00:44.318796 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:44.318588 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4npnf\" (UniqueName: \"kubernetes.io/projected/ee5dbe45-5126-4c3e-9397-69e2153637de-kube-api-access-4npnf\") pod \"console-765cb78b68-dqp8h\" (UID: \"ee5dbe45-5126-4c3e-9397-69e2153637de\") " pod="openshift-console/console-765cb78b68-dqp8h" Apr 16 14:00:44.319196 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:44.319165 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ee5dbe45-5126-4c3e-9397-69e2153637de-oauth-serving-cert\") pod \"console-765cb78b68-dqp8h\" (UID: \"ee5dbe45-5126-4c3e-9397-69e2153637de\") " pod="openshift-console/console-765cb78b68-dqp8h" Apr 16 14:00:44.319412 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:44.319385 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ee5dbe45-5126-4c3e-9397-69e2153637de-service-ca\") pod \"console-765cb78b68-dqp8h\" (UID: \"ee5dbe45-5126-4c3e-9397-69e2153637de\") " pod="openshift-console/console-765cb78b68-dqp8h" Apr 16 14:00:44.319652 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:44.319549 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ee5dbe45-5126-4c3e-9397-69e2153637de-console-config\") pod \"console-765cb78b68-dqp8h\" (UID: \"ee5dbe45-5126-4c3e-9397-69e2153637de\") " pod="openshift-console/console-765cb78b68-dqp8h" Apr 16 14:00:44.321307 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:44.321283 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ee5dbe45-5126-4c3e-9397-69e2153637de-console-serving-cert\") pod \"console-765cb78b68-dqp8h\" (UID: \"ee5dbe45-5126-4c3e-9397-69e2153637de\") " pod="openshift-console/console-765cb78b68-dqp8h" Apr 16 14:00:44.321464 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:44.321441 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ee5dbe45-5126-4c3e-9397-69e2153637de-console-oauth-config\") pod \"console-765cb78b68-dqp8h\" (UID: \"ee5dbe45-5126-4c3e-9397-69e2153637de\") " pod="openshift-console/console-765cb78b68-dqp8h" Apr 16 14:00:44.326872 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:44.326851 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4npnf\" (UniqueName: \"kubernetes.io/projected/ee5dbe45-5126-4c3e-9397-69e2153637de-kube-api-access-4npnf\") pod \"console-765cb78b68-dqp8h\" (UID: \"ee5dbe45-5126-4c3e-9397-69e2153637de\") " pod="openshift-console/console-765cb78b68-dqp8h" Apr 16 14:00:44.405838 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:44.405777 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-765cb78b68-dqp8h" Apr 16 14:00:44.543428 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:44.543391 2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-765cb78b68-dqp8h"] Apr 16 14:00:44.547604 ip-10-0-136-114 kubenswrapper[2574]: W0416 14:00:44.547573 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee5dbe45_5126_4c3e_9397_69e2153637de.slice/crio-9f5ee13d62e45f85237ead42e1095105a9efda4fcc5b2871a61058348d187b3d WatchSource:0}: Error finding container 9f5ee13d62e45f85237ead42e1095105a9efda4fcc5b2871a61058348d187b3d: Status 404 returned error can't find the container with id 9f5ee13d62e45f85237ead42e1095105a9efda4fcc5b2871a61058348d187b3d Apr 16 14:00:45.363886 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:45.363847 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-765cb78b68-dqp8h" event={"ID":"ee5dbe45-5126-4c3e-9397-69e2153637de","Type":"ContainerStarted","Data":"9f5ee13d62e45f85237ead42e1095105a9efda4fcc5b2871a61058348d187b3d"} Apr 16 14:00:48.374816 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:48.374776 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-765cb78b68-dqp8h" event={"ID":"ee5dbe45-5126-4c3e-9397-69e2153637de","Type":"ContainerStarted","Data":"213e645aaf8882db06ed30228245ac7b98e48f65c2e50c3193aa6d3db5d28194"} Apr 16 14:00:48.394474 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:48.394423 2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-765cb78b68-dqp8h" podStartSLOduration=0.741133059 podStartE2EDuration="4.394408793s" podCreationTimestamp="2026-04-16 14:00:44 +0000 UTC" firstStartedPulling="2026-04-16 14:00:44.549878713 +0000 UTC m=+214.379613560" lastFinishedPulling="2026-04-16 14:00:48.203154431 +0000 UTC m=+218.032889294" observedRunningTime="2026-04-16 14:00:48.392579246 +0000 UTC m=+218.222314167" watchObservedRunningTime="2026-04-16 14:00:48.394408793 +0000 UTC m=+218.224143664" Apr 16 14:00:52.482402 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:52.482366 2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-6d646bb8f8-wzpkq"] Apr 16 14:00:52.485867 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:52.485847 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d646bb8f8-wzpkq" Apr 16 14:00:52.493356 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:52.493154 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Apr 16 14:00:52.494695 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:52.494219 2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d646bb8f8-wzpkq"] Apr 16 14:00:52.597323 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:52.597280 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4e213a29-44bc-448b-a4e4-b77a596aeb54-console-serving-cert\") pod \"console-6d646bb8f8-wzpkq\" (UID: \"4e213a29-44bc-448b-a4e4-b77a596aeb54\") " pod="openshift-console/console-6d646bb8f8-wzpkq" Apr 16 14:00:52.597496 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:52.597333 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4e213a29-44bc-448b-a4e4-b77a596aeb54-console-oauth-config\") pod \"console-6d646bb8f8-wzpkq\" (UID: \"4e213a29-44bc-448b-a4e4-b77a596aeb54\") " pod="openshift-console/console-6d646bb8f8-wzpkq" Apr 16 14:00:52.597496 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:52.597363 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjjr4\" (UniqueName: \"kubernetes.io/projected/4e213a29-44bc-448b-a4e4-b77a596aeb54-kube-api-access-sjjr4\") pod \"console-6d646bb8f8-wzpkq\" (UID: \"4e213a29-44bc-448b-a4e4-b77a596aeb54\") " pod="openshift-console/console-6d646bb8f8-wzpkq" Apr 16 14:00:52.597496 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:52.597485 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4e213a29-44bc-448b-a4e4-b77a596aeb54-console-config\") pod \"console-6d646bb8f8-wzpkq\" (UID: \"4e213a29-44bc-448b-a4e4-b77a596aeb54\") " pod="openshift-console/console-6d646bb8f8-wzpkq" Apr 16 14:00:52.597637 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:52.597531 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4e213a29-44bc-448b-a4e4-b77a596aeb54-service-ca\") pod \"console-6d646bb8f8-wzpkq\" (UID: \"4e213a29-44bc-448b-a4e4-b77a596aeb54\") " pod="openshift-console/console-6d646bb8f8-wzpkq" Apr 16 14:00:52.597637 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:52.597592 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4e213a29-44bc-448b-a4e4-b77a596aeb54-oauth-serving-cert\") pod \"console-6d646bb8f8-wzpkq\" (UID: \"4e213a29-44bc-448b-a4e4-b77a596aeb54\") " pod="openshift-console/console-6d646bb8f8-wzpkq" Apr 16 14:00:52.597719 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:52.597642 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e213a29-44bc-448b-a4e4-b77a596aeb54-trusted-ca-bundle\") pod \"console-6d646bb8f8-wzpkq\" (UID: \"4e213a29-44bc-448b-a4e4-b77a596aeb54\") " pod="openshift-console/console-6d646bb8f8-wzpkq" Apr 16 14:00:52.698924 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:52.698888 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e213a29-44bc-448b-a4e4-b77a596aeb54-trusted-ca-bundle\") pod \"console-6d646bb8f8-wzpkq\" (UID: \"4e213a29-44bc-448b-a4e4-b77a596aeb54\") " pod="openshift-console/console-6d646bb8f8-wzpkq" Apr 16 14:00:52.699109 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:52.698948 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4e213a29-44bc-448b-a4e4-b77a596aeb54-console-serving-cert\") pod \"console-6d646bb8f8-wzpkq\" (UID: \"4e213a29-44bc-448b-a4e4-b77a596aeb54\") " pod="openshift-console/console-6d646bb8f8-wzpkq" Apr 16 14:00:52.699109 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:52.698977 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4e213a29-44bc-448b-a4e4-b77a596aeb54-console-oauth-config\") pod \"console-6d646bb8f8-wzpkq\" (UID: \"4e213a29-44bc-448b-a4e4-b77a596aeb54\") " pod="openshift-console/console-6d646bb8f8-wzpkq" Apr 16 14:00:52.699109 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:52.699000 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-sjjr4\" (UniqueName: \"kubernetes.io/projected/4e213a29-44bc-448b-a4e4-b77a596aeb54-kube-api-access-sjjr4\") pod \"console-6d646bb8f8-wzpkq\" (UID: \"4e213a29-44bc-448b-a4e4-b77a596aeb54\") " pod="openshift-console/console-6d646bb8f8-wzpkq" Apr 16 14:00:52.699109 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:52.699066 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4e213a29-44bc-448b-a4e4-b77a596aeb54-console-config\") pod \"console-6d646bb8f8-wzpkq\" (UID: \"4e213a29-44bc-448b-a4e4-b77a596aeb54\") " pod="openshift-console/console-6d646bb8f8-wzpkq" Apr 16 14:00:52.699109 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:52.699103 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4e213a29-44bc-448b-a4e4-b77a596aeb54-service-ca\") pod \"console-6d646bb8f8-wzpkq\" (UID: \"4e213a29-44bc-448b-a4e4-b77a596aeb54\") " pod="openshift-console/console-6d646bb8f8-wzpkq" Apr 16 14:00:52.699369 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:52.699145 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4e213a29-44bc-448b-a4e4-b77a596aeb54-oauth-serving-cert\") pod \"console-6d646bb8f8-wzpkq\" (UID: \"4e213a29-44bc-448b-a4e4-b77a596aeb54\") " pod="openshift-console/console-6d646bb8f8-wzpkq" Apr 16 14:00:52.699953 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:52.699922 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e213a29-44bc-448b-a4e4-b77a596aeb54-trusted-ca-bundle\") pod \"console-6d646bb8f8-wzpkq\" (UID: \"4e213a29-44bc-448b-a4e4-b77a596aeb54\") " pod="openshift-console/console-6d646bb8f8-wzpkq" Apr 16 14:00:52.700168 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:52.700119 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4e213a29-44bc-448b-a4e4-b77a596aeb54-oauth-serving-cert\") pod \"console-6d646bb8f8-wzpkq\" (UID: \"4e213a29-44bc-448b-a4e4-b77a596aeb54\") " pod="openshift-console/console-6d646bb8f8-wzpkq" Apr 16 14:00:52.700287 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:52.700186 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4e213a29-44bc-448b-a4e4-b77a596aeb54-console-config\") pod \"console-6d646bb8f8-wzpkq\" (UID: \"4e213a29-44bc-448b-a4e4-b77a596aeb54\") " pod="openshift-console/console-6d646bb8f8-wzpkq" Apr 16 14:00:52.700353 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:52.700335 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4e213a29-44bc-448b-a4e4-b77a596aeb54-service-ca\") pod \"console-6d646bb8f8-wzpkq\" (UID: \"4e213a29-44bc-448b-a4e4-b77a596aeb54\") " pod="openshift-console/console-6d646bb8f8-wzpkq" Apr 16 14:00:52.701726 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:52.701701 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4e213a29-44bc-448b-a4e4-b77a596aeb54-console-oauth-config\") pod \"console-6d646bb8f8-wzpkq\" (UID: \"4e213a29-44bc-448b-a4e4-b77a596aeb54\") " pod="openshift-console/console-6d646bb8f8-wzpkq" Apr 16 14:00:52.701950 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:52.701931 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4e213a29-44bc-448b-a4e4-b77a596aeb54-console-serving-cert\") pod \"console-6d646bb8f8-wzpkq\" (UID: \"4e213a29-44bc-448b-a4e4-b77a596aeb54\") " pod="openshift-console/console-6d646bb8f8-wzpkq" Apr 16 14:00:52.706247 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:52.706228 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjjr4\" (UniqueName: \"kubernetes.io/projected/4e213a29-44bc-448b-a4e4-b77a596aeb54-kube-api-access-sjjr4\") pod \"console-6d646bb8f8-wzpkq\" (UID: \"4e213a29-44bc-448b-a4e4-b77a596aeb54\") " pod="openshift-console/console-6d646bb8f8-wzpkq" Apr 16 14:00:52.797562 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:52.797474 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d646bb8f8-wzpkq" Apr 16 14:00:54.405957 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:54.405916 2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-765cb78b68-dqp8h" Apr 16 14:00:54.406391 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:54.405969 2574 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-765cb78b68-dqp8h" Apr 16 14:00:54.411217 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:54.411190 2574 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-765cb78b68-dqp8h" Apr 16 14:00:55.399440 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:55.399407 2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-765cb78b68-dqp8h" Apr 16 14:00:56.550867 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:56.550844 2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d646bb8f8-wzpkq"] Apr 16 14:00:56.554126 ip-10-0-136-114 kubenswrapper[2574]: W0416 14:00:56.554096 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e213a29_44bc_448b_a4e4_b77a596aeb54.slice/crio-64b48d1dfb57e77b457d54cbda61963ddc4efe363119b9d23a401996f91c3e55 WatchSource:0}: Error finding container 64b48d1dfb57e77b457d54cbda61963ddc4efe363119b9d23a401996f91c3e55: Status 404 returned error can't find the container with id 64b48d1dfb57e77b457d54cbda61963ddc4efe363119b9d23a401996f91c3e55 Apr 16 14:00:57.402816 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:57.402774 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-586b57c7b4-bd2j9" event={"ID":"9b1f9023-4bfb-4ee6-9045-780848f02427","Type":"ContainerStarted","Data":"36a95a245bb07eec28b910103f7950c9745e7185f4a684923df35c3ca01d2ac1"} Apr 16 14:00:57.403006 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:57.402949 2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-586b57c7b4-bd2j9" Apr 16 14:00:57.404437 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:57.404407 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d646bb8f8-wzpkq" event={"ID":"4e213a29-44bc-448b-a4e4-b77a596aeb54","Type":"ContainerStarted","Data":"16f64eebb681da7a9420c4320890641e6b816a9ca68c90db5c768e2eaa0d3705"} Apr 16 14:00:57.404437 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:57.404437 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d646bb8f8-wzpkq" event={"ID":"4e213a29-44bc-448b-a4e4-b77a596aeb54","Type":"ContainerStarted","Data":"64b48d1dfb57e77b457d54cbda61963ddc4efe363119b9d23a401996f91c3e55"} Apr 16 14:00:57.405860 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:57.405832 2574 generic.go:358] "Generic (PLEG): container finished" podID="e9f0f164-4788-43c4-890f-52ed0197b594" containerID="fdfae3918d93a4e1f86251f7f227aff4a3aee07165e13d97032d61e32b18aa1e" exitCode=0 Apr 16 14:00:57.405961 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:57.405890 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-69965bb79d-hzmxv" event={"ID":"e9f0f164-4788-43c4-890f-52ed0197b594","Type":"ContainerDied","Data":"fdfae3918d93a4e1f86251f7f227aff4a3aee07165e13d97032d61e32b18aa1e"} Apr 16 14:00:57.406274 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:57.406256 2574 scope.go:117] "RemoveContainer" containerID="fdfae3918d93a4e1f86251f7f227aff4a3aee07165e13d97032d61e32b18aa1e" Apr 16 14:00:57.415415 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:57.415395 2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-586b57c7b4-bd2j9" Apr 16 14:00:57.421428 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:57.421383 2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/downloads-586b57c7b4-bd2j9" podStartSLOduration=2.29505778 podStartE2EDuration="18.421370067s" podCreationTimestamp="2026-04-16 14:00:39 +0000 UTC" firstStartedPulling="2026-04-16 14:00:40.409079954 +0000 UTC m=+210.238814801" lastFinishedPulling="2026-04-16 14:00:56.535392237 +0000 UTC m=+226.365127088" observedRunningTime="2026-04-16 14:00:57.418866543 +0000 UTC m=+227.248601414" watchObservedRunningTime="2026-04-16 14:00:57.421370067 +0000 UTC m=+227.251104938" Apr 16 14:00:57.436345 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:57.436287 2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-6d646bb8f8-wzpkq" podStartSLOduration=5.436274316 podStartE2EDuration="5.436274316s" podCreationTimestamp="2026-04-16 14:00:52 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 14:00:57.436010351 +0000 UTC m=+227.265745222" watchObservedRunningTime="2026-04-16 14:00:57.436274316 +0000 UTC m=+227.266009179" Apr 16 14:00:58.410931 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:58.410886 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-69965bb79d-hzmxv" event={"ID":"e9f0f164-4788-43c4-890f-52ed0197b594","Type":"ContainerStarted","Data":"10234e6074e2a66db653ca025e0c330a144edf0f2ff710f29da79405b7b48dfa"} Apr 16 14:00:59.136129 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:59.136069 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1674f1e9-d0f3-49cf-9c20-40d72e9428e9/init-config-reloader/0.log" Apr 16 14:00:59.335878 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:59.335846 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1674f1e9-d0f3-49cf-9c20-40d72e9428e9/alertmanager/0.log" Apr 16 14:00:59.535424 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:59.535347 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1674f1e9-d0f3-49cf-9c20-40d72e9428e9/config-reloader/0.log" Apr 16 14:00:59.735892 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:59.735864 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1674f1e9-d0f3-49cf-9c20-40d72e9428e9/kube-rbac-proxy-web/0.log" Apr 16 14:00:59.935648 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:00:59.935618 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1674f1e9-d0f3-49cf-9c20-40d72e9428e9/kube-rbac-proxy/0.log" Apr 16 14:01:00.136071 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:00.136042 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1674f1e9-d0f3-49cf-9c20-40d72e9428e9/kube-rbac-proxy-metric/0.log" Apr 16 14:01:00.335195 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:00.335167 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1674f1e9-d0f3-49cf-9c20-40d72e9428e9/prom-label-proxy/0.log" Apr 16 14:01:00.537325 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:00.537294 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6667474d89-tj5v6_5b9fe9ed-1a79-4590-ad6e-74acb8e02aed/cluster-monitoring-operator/0.log" Apr 16 14:01:00.735884 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:00.735800 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7479c89684-8xh94_74619757-38ff-48ef-9c0c-129c39811a4d/kube-state-metrics/0.log" Apr 16 14:01:00.935098 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:00.935065 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7479c89684-8xh94_74619757-38ff-48ef-9c0c-129c39811a4d/kube-rbac-proxy-main/0.log" Apr 16 14:01:01.135363 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:01.135338 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7479c89684-8xh94_74619757-38ff-48ef-9c0c-129c39811a4d/kube-rbac-proxy-self/0.log" Apr 16 14:01:02.344049 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:02.344021 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h7mfj_35436df9-1548-4b04-9e65-ca60b80486e7/init-textfile/0.log" Apr 16 14:01:02.539672 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:02.539643 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h7mfj_35436df9-1548-4b04-9e65-ca60b80486e7/node-exporter/0.log" Apr 16 14:01:02.740531 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:02.740449 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h7mfj_35436df9-1548-4b04-9e65-ca60b80486e7/kube-rbac-proxy/0.log" Apr 16 14:01:02.797859 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:02.797826 2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6d646bb8f8-wzpkq" Apr 16 14:01:02.798027 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:02.797879 2574 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6d646bb8f8-wzpkq" Apr 16 14:01:02.803283 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:02.803254 2574 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6d646bb8f8-wzpkq" Apr 16 14:01:03.429307 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:03.429278 2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6d646bb8f8-wzpkq" Apr 16 14:01:03.482447 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:03.482410 2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-765cb78b68-dqp8h"] Apr 16 14:01:03.539678 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:03.539652 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-5669946b84-q57gd_0d969860-3250-40b5-b124-a01d0c99a8a1/kube-rbac-proxy-main/0.log" Apr 16 14:01:03.739625 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:03.739554 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-5669946b84-q57gd_0d969860-3250-40b5-b124-a01d0c99a8a1/kube-rbac-proxy-self/0.log" Apr 16 14:01:03.935635 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:03.935588 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-5669946b84-q57gd_0d969860-3250-40b5-b124-a01d0c99a8a1/openshift-state-metrics/0.log" Apr 16 14:01:05.935484 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:05.935455 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-9cb97cd87-cnfb5_4f118646-7573-4184-880d-eb313a7b063d/prometheus-operator-admission-webhook/0.log" Apr 16 14:01:06.135303 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:06.135269 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c94669ccd-k5vvr_78e0f4b8-9966-439a-98af-e8d7a5253a42/thanos-query/0.log" Apr 16 14:01:06.336176 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:06.336135 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c94669ccd-k5vvr_78e0f4b8-9966-439a-98af-e8d7a5253a42/kube-rbac-proxy-web/0.log" Apr 16 14:01:06.535044 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:06.534987 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c94669ccd-k5vvr_78e0f4b8-9966-439a-98af-e8d7a5253a42/kube-rbac-proxy/0.log" Apr 16 14:01:06.735492 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:06.735410 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c94669ccd-k5vvr_78e0f4b8-9966-439a-98af-e8d7a5253a42/prom-label-proxy/0.log" Apr 16 14:01:06.935899 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:06.935871 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c94669ccd-k5vvr_78e0f4b8-9966-439a-98af-e8d7a5253a42/kube-rbac-proxy-rules/0.log" Apr 16 14:01:07.135067 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:07.135034 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c94669ccd-k5vvr_78e0f4b8-9966-439a-98af-e8d7a5253a42/kube-rbac-proxy-metrics/0.log" Apr 16 14:01:07.935525 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:07.935499 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6d646bb8f8-wzpkq_4e213a29-44bc-448b-a4e4-b77a596aeb54/console/0.log" Apr 16 14:01:08.138718 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:08.138686 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-765cb78b68-dqp8h_ee5dbe45-5126-4c3e-9397-69e2153637de/console/0.log" Apr 16 14:01:08.336434 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:08.336409 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-586b57c7b4-bd2j9_9b1f9023-4bfb-4ee6-9045-780848f02427/download-server/0.log" Apr 16 14:01:10.135536 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:10.135506 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-x4qcz_c590b6a7-f5a5-476f-a0bd-fb8e8853b8cb/dns-node-resolver/0.log" Apr 16 14:01:21.468486 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:21.468448 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e-metrics-certs\") pod \"network-metrics-daemon-5fh28\" (UID: \"2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e\") " pod="openshift-multus/network-metrics-daemon-5fh28" Apr 16 14:01:21.470720 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:21.470692 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e-metrics-certs\") pod \"network-metrics-daemon-5fh28\" (UID: \"2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e\") " pod="openshift-multus/network-metrics-daemon-5fh28" Apr 16 14:01:21.609400 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:21.609364 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jz268\"" Apr 16 14:01:21.617219 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:21.617201 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5fh28" Apr 16 14:01:21.731468 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:21.731401 2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5fh28"] Apr 16 14:01:21.735046 ip-10-0-136-114 kubenswrapper[2574]: W0416 14:01:21.735017 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eb1b9a0_87a2_478e_a0ea_b2da8ceced2e.slice/crio-6a044065870b543efafc824be42570976d45f43ad33a1f1347621e3fe29804dd WatchSource:0}: Error finding container 6a044065870b543efafc824be42570976d45f43ad33a1f1347621e3fe29804dd: Status 404 returned error can't find the container with id 6a044065870b543efafc824be42570976d45f43ad33a1f1347621e3fe29804dd Apr 16 14:01:22.484088 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:22.484005 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5fh28" event={"ID":"2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e","Type":"ContainerStarted","Data":"6a044065870b543efafc824be42570976d45f43ad33a1f1347621e3fe29804dd"} Apr 16 14:01:24.491457 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:24.491418 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5fh28" event={"ID":"2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e","Type":"ContainerStarted","Data":"ca8263c4bc6c65a296360cfbc7e6d913f0c7976fb2a7bbd9160987a101e77df5"} Apr 16 14:01:24.491457 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:24.491460 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5fh28" event={"ID":"2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e","Type":"ContainerStarted","Data":"4332a8536aeac5132b80623628215395bf1d8f1ab32f6e79a6b52679c7d00b3c"} Apr 16 14:01:24.515950 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:24.515893 2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-5fh28" podStartSLOduration=252.831642936 podStartE2EDuration="4m14.515874689s" podCreationTimestamp="2026-04-16 13:57:10 +0000 UTC" firstStartedPulling="2026-04-16 14:01:21.737076839 +0000 UTC m=+251.566811692" lastFinishedPulling="2026-04-16 14:01:23.421308584 +0000 UTC m=+253.251043445" observedRunningTime="2026-04-16 14:01:24.514536768 +0000 UTC m=+254.344271638" watchObservedRunningTime="2026-04-16 14:01:24.515874689 +0000 UTC m=+254.345609565" Apr 16 14:01:28.505703 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:28.505665 2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-765cb78b68-dqp8h" podUID="ee5dbe45-5126-4c3e-9397-69e2153637de" containerName="console" containerID="cri-o://213e645aaf8882db06ed30228245ac7b98e48f65c2e50c3193aa6d3db5d28194" gracePeriod=15 Apr 16 14:01:28.782380 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:28.782355 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-765cb78b68-dqp8h_ee5dbe45-5126-4c3e-9397-69e2153637de/console/0.log" Apr 16 14:01:28.782530 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:28.782425 2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-765cb78b68-dqp8h" Apr 16 14:01:28.829000 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:28.828976 2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4npnf\" (UniqueName: \"kubernetes.io/projected/ee5dbe45-5126-4c3e-9397-69e2153637de-kube-api-access-4npnf\") pod \"ee5dbe45-5126-4c3e-9397-69e2153637de\" (UID: \"ee5dbe45-5126-4c3e-9397-69e2153637de\") " Apr 16 14:01:28.829132 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:28.829007 2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ee5dbe45-5126-4c3e-9397-69e2153637de-oauth-serving-cert\") pod \"ee5dbe45-5126-4c3e-9397-69e2153637de\" (UID: \"ee5dbe45-5126-4c3e-9397-69e2153637de\") " Apr 16 14:01:28.829132 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:28.829039 2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ee5dbe45-5126-4c3e-9397-69e2153637de-console-oauth-config\") pod \"ee5dbe45-5126-4c3e-9397-69e2153637de\" (UID: \"ee5dbe45-5126-4c3e-9397-69e2153637de\") " Apr 16 14:01:28.829132 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:28.829062 2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ee5dbe45-5126-4c3e-9397-69e2153637de-service-ca\") pod \"ee5dbe45-5126-4c3e-9397-69e2153637de\" (UID: \"ee5dbe45-5126-4c3e-9397-69e2153637de\") " Apr 16 14:01:28.829287 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:28.829171 2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ee5dbe45-5126-4c3e-9397-69e2153637de-console-serving-cert\") pod \"ee5dbe45-5126-4c3e-9397-69e2153637de\" (UID: \"ee5dbe45-5126-4c3e-9397-69e2153637de\") " Apr 16 14:01:28.829287 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:28.829274 2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ee5dbe45-5126-4c3e-9397-69e2153637de-console-config\") pod \"ee5dbe45-5126-4c3e-9397-69e2153637de\" (UID: \"ee5dbe45-5126-4c3e-9397-69e2153637de\") " Apr 16 14:01:28.829447 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:28.829416 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee5dbe45-5126-4c3e-9397-69e2153637de-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ee5dbe45-5126-4c3e-9397-69e2153637de" (UID: "ee5dbe45-5126-4c3e-9397-69e2153637de"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 16 14:01:28.829552 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:28.829521 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee5dbe45-5126-4c3e-9397-69e2153637de-service-ca" (OuterVolumeSpecName: "service-ca") pod "ee5dbe45-5126-4c3e-9397-69e2153637de" (UID: "ee5dbe45-5126-4c3e-9397-69e2153637de"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 16 14:01:28.829687 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:28.829625 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee5dbe45-5126-4c3e-9397-69e2153637de-console-config" (OuterVolumeSpecName: "console-config") pod "ee5dbe45-5126-4c3e-9397-69e2153637de" (UID: "ee5dbe45-5126-4c3e-9397-69e2153637de"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 16 14:01:28.829774 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:28.829711 2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ee5dbe45-5126-4c3e-9397-69e2153637de-service-ca\") on node \"ip-10-0-136-114.ec2.internal\" DevicePath \"\"" Apr 16 14:01:28.829774 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:28.829731 2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ee5dbe45-5126-4c3e-9397-69e2153637de-console-config\") on node \"ip-10-0-136-114.ec2.internal\" DevicePath \"\"" Apr 16 14:01:28.829774 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:28.829763 2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ee5dbe45-5126-4c3e-9397-69e2153637de-oauth-serving-cert\") on node \"ip-10-0-136-114.ec2.internal\" DevicePath \"\"" Apr 16 14:01:28.831265 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:28.831242 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee5dbe45-5126-4c3e-9397-69e2153637de-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ee5dbe45-5126-4c3e-9397-69e2153637de" (UID: "ee5dbe45-5126-4c3e-9397-69e2153637de"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 16 14:01:28.831536 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:28.831520 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee5dbe45-5126-4c3e-9397-69e2153637de-kube-api-access-4npnf" (OuterVolumeSpecName: "kube-api-access-4npnf") pod "ee5dbe45-5126-4c3e-9397-69e2153637de" (UID: "ee5dbe45-5126-4c3e-9397-69e2153637de"). InnerVolumeSpecName "kube-api-access-4npnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 16 14:01:28.831657 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:28.831637 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee5dbe45-5126-4c3e-9397-69e2153637de-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ee5dbe45-5126-4c3e-9397-69e2153637de" (UID: "ee5dbe45-5126-4c3e-9397-69e2153637de"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 16 14:01:28.930361 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:28.930327 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4npnf\" (UniqueName: \"kubernetes.io/projected/ee5dbe45-5126-4c3e-9397-69e2153637de-kube-api-access-4npnf\") on node \"ip-10-0-136-114.ec2.internal\" DevicePath \"\"" Apr 16 14:01:28.930361 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:28.930356 2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ee5dbe45-5126-4c3e-9397-69e2153637de-console-oauth-config\") on node \"ip-10-0-136-114.ec2.internal\" DevicePath \"\"" Apr 16 14:01:28.930361 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:28.930367 2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ee5dbe45-5126-4c3e-9397-69e2153637de-console-serving-cert\") on node \"ip-10-0-136-114.ec2.internal\" DevicePath \"\"" Apr 16 14:01:29.506409 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:29.506381 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-765cb78b68-dqp8h_ee5dbe45-5126-4c3e-9397-69e2153637de/console/0.log" Apr 16 14:01:29.506835 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:29.506422 2574 generic.go:358] "Generic (PLEG): container finished" podID="ee5dbe45-5126-4c3e-9397-69e2153637de" containerID="213e645aaf8882db06ed30228245ac7b98e48f65c2e50c3193aa6d3db5d28194" exitCode=2 Apr 16 14:01:29.506835 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:29.506493 2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-765cb78b68-dqp8h" Apr 16 14:01:29.506835 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:29.506510 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-765cb78b68-dqp8h" event={"ID":"ee5dbe45-5126-4c3e-9397-69e2153637de","Type":"ContainerDied","Data":"213e645aaf8882db06ed30228245ac7b98e48f65c2e50c3193aa6d3db5d28194"} Apr 16 14:01:29.506835 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:29.506553 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-765cb78b68-dqp8h" event={"ID":"ee5dbe45-5126-4c3e-9397-69e2153637de","Type":"ContainerDied","Data":"9f5ee13d62e45f85237ead42e1095105a9efda4fcc5b2871a61058348d187b3d"} Apr 16 14:01:29.506835 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:29.506573 2574 scope.go:117] "RemoveContainer" containerID="213e645aaf8882db06ed30228245ac7b98e48f65c2e50c3193aa6d3db5d28194" Apr 16 14:01:29.522265 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:29.522250 2574 scope.go:117] "RemoveContainer" containerID="213e645aaf8882db06ed30228245ac7b98e48f65c2e50c3193aa6d3db5d28194" Apr 16 14:01:29.522514 ip-10-0-136-114 kubenswrapper[2574]: E0416 14:01:29.522498 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"213e645aaf8882db06ed30228245ac7b98e48f65c2e50c3193aa6d3db5d28194\": container with ID starting with 213e645aaf8882db06ed30228245ac7b98e48f65c2e50c3193aa6d3db5d28194 not found: ID does not exist" containerID="213e645aaf8882db06ed30228245ac7b98e48f65c2e50c3193aa6d3db5d28194" Apr 16 14:01:29.522553 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:29.522521 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"213e645aaf8882db06ed30228245ac7b98e48f65c2e50c3193aa6d3db5d28194"} err="failed to get container status \"213e645aaf8882db06ed30228245ac7b98e48f65c2e50c3193aa6d3db5d28194\": rpc error: code = NotFound desc = could not find container \"213e645aaf8882db06ed30228245ac7b98e48f65c2e50c3193aa6d3db5d28194\": container with ID starting with 213e645aaf8882db06ed30228245ac7b98e48f65c2e50c3193aa6d3db5d28194 not found: ID does not exist" Apr 16 14:01:29.527616 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:29.527594 2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-765cb78b68-dqp8h"] Apr 16 14:01:29.532019 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:29.532001 2574 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-765cb78b68-dqp8h"] Apr 16 14:01:30.809698 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:30.809655 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee5dbe45-5126-4c3e-9397-69e2153637de" path="/var/lib/kubelet/pods/ee5dbe45-5126-4c3e-9397-69e2153637de/volumes" Apr 16 14:01:45.557942 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:45.557867 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6667474d89-tj5v6_5b9fe9ed-1a79-4590-ad6e-74acb8e02aed/cluster-monitoring-operator/0.log" Apr 16 14:01:45.557942 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:45.557916 2574 generic.go:358] "Generic (PLEG): container finished" podID="5b9fe9ed-1a79-4590-ad6e-74acb8e02aed" containerID="1d8338b5ca3154b7fae96c1a22402a026c5a00ac22ac996a2f97b419aadaa8c0" exitCode=2 Apr 16 14:01:45.558371 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:45.557968 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6667474d89-tj5v6" event={"ID":"5b9fe9ed-1a79-4590-ad6e-74acb8e02aed","Type":"ContainerDied","Data":"1d8338b5ca3154b7fae96c1a22402a026c5a00ac22ac996a2f97b419aadaa8c0"} Apr 16 14:01:45.558371 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:45.558319 2574 scope.go:117] "RemoveContainer" containerID="1d8338b5ca3154b7fae96c1a22402a026c5a00ac22ac996a2f97b419aadaa8c0" Apr 16 14:01:46.562603 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:46.562575 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6667474d89-tj5v6_5b9fe9ed-1a79-4590-ad6e-74acb8e02aed/cluster-monitoring-operator/0.log" Apr 16 14:01:46.563007 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:46.562645 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6667474d89-tj5v6" event={"ID":"5b9fe9ed-1a79-4590-ad6e-74acb8e02aed","Type":"ContainerStarted","Data":"cb7635d19444f6f7ec8608bf5e5741fabf92f3d706a0fd7e913dd80b464b622e"} Apr 16 14:01:50.082257 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:50.082217 2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Apr 16 14:01:50.082849 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:50.082822 2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1674f1e9-d0f3-49cf-9c20-40d72e9428e9" containerName="alertmanager" containerID="cri-o://c98b6e56106018578f36bcfd1e52ce2f0a937c05220ba15225e72d68577b5675" gracePeriod=120 Apr 16 14:01:50.082958 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:50.082887 2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1674f1e9-d0f3-49cf-9c20-40d72e9428e9" containerName="kube-rbac-proxy-web" containerID="cri-o://9c1100377039bff38551f96b0a7bb53031da5e82e496d8d6f7e56e48d24a9c89" gracePeriod=120 Apr 16 14:01:50.083018 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:50.082923 2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1674f1e9-d0f3-49cf-9c20-40d72e9428e9" containerName="config-reloader" containerID="cri-o://2a98591981bae2c95137ab5cb7fa07f6e15c6fd690e1d01717204617f5248513" gracePeriod=120 Apr 16 14:01:50.083069 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:50.082869 2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1674f1e9-d0f3-49cf-9c20-40d72e9428e9" containerName="kube-rbac-proxy-metric" containerID="cri-o://57ef3cdb8657c272c6327739170ad1fc5a64a07e80709e961b8c458ec9cbbe51" gracePeriod=120 Apr 16 14:01:50.083069 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:50.082943 2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1674f1e9-d0f3-49cf-9c20-40d72e9428e9" containerName="kube-rbac-proxy" containerID="cri-o://8a107c531e665503ab11c11b3c3393f93b4baf07bed584f9d77e9b0b398d2400" gracePeriod=120 Apr 16 14:01:50.083162 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:50.083066 2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1674f1e9-d0f3-49cf-9c20-40d72e9428e9" containerName="prom-label-proxy" containerID="cri-o://cc125dfb019a0fc9f2aa8b910005d58e61f61ca47f96df371db6bd86f9c08d0f" gracePeriod=120 Apr 16 14:01:50.576567 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:50.576528 2574 generic.go:358] "Generic (PLEG): container finished" podID="1674f1e9-d0f3-49cf-9c20-40d72e9428e9" containerID="cc125dfb019a0fc9f2aa8b910005d58e61f61ca47f96df371db6bd86f9c08d0f" exitCode=0 Apr 16 14:01:50.576567 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:50.576556 2574 generic.go:358] "Generic (PLEG): container finished" podID="1674f1e9-d0f3-49cf-9c20-40d72e9428e9" containerID="57ef3cdb8657c272c6327739170ad1fc5a64a07e80709e961b8c458ec9cbbe51" exitCode=0 Apr 16 14:01:50.576567 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:50.576562 2574 generic.go:358] "Generic (PLEG): container finished" podID="1674f1e9-d0f3-49cf-9c20-40d72e9428e9" containerID="8a107c531e665503ab11c11b3c3393f93b4baf07bed584f9d77e9b0b398d2400" exitCode=0 Apr 16 14:01:50.576567 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:50.576568 2574 generic.go:358] "Generic (PLEG): container finished" podID="1674f1e9-d0f3-49cf-9c20-40d72e9428e9" containerID="2a98591981bae2c95137ab5cb7fa07f6e15c6fd690e1d01717204617f5248513" exitCode=0 Apr 16 14:01:50.576567 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:50.576575 2574 generic.go:358] "Generic (PLEG): container finished" podID="1674f1e9-d0f3-49cf-9c20-40d72e9428e9" containerID="c98b6e56106018578f36bcfd1e52ce2f0a937c05220ba15225e72d68577b5675" exitCode=0 Apr 16 14:01:50.576912 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:50.576602 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1674f1e9-d0f3-49cf-9c20-40d72e9428e9","Type":"ContainerDied","Data":"cc125dfb019a0fc9f2aa8b910005d58e61f61ca47f96df371db6bd86f9c08d0f"} Apr 16 14:01:50.576912 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:50.576646 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1674f1e9-d0f3-49cf-9c20-40d72e9428e9","Type":"ContainerDied","Data":"57ef3cdb8657c272c6327739170ad1fc5a64a07e80709e961b8c458ec9cbbe51"} Apr 16 14:01:50.576912 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:50.576669 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1674f1e9-d0f3-49cf-9c20-40d72e9428e9","Type":"ContainerDied","Data":"8a107c531e665503ab11c11b3c3393f93b4baf07bed584f9d77e9b0b398d2400"} Apr 16 14:01:50.576912 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:50.576682 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1674f1e9-d0f3-49cf-9c20-40d72e9428e9","Type":"ContainerDied","Data":"2a98591981bae2c95137ab5cb7fa07f6e15c6fd690e1d01717204617f5248513"} Apr 16 14:01:50.576912 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:50.576693 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1674f1e9-d0f3-49cf-9c20-40d72e9428e9","Type":"ContainerDied","Data":"c98b6e56106018578f36bcfd1e52ce2f0a937c05220ba15225e72d68577b5675"} Apr 16 14:01:51.314563 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.314543 2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:01:51.429195 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.429110 2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8sbs9\" (UniqueName: \"kubernetes.io/projected/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-kube-api-access-8sbs9\") pod \"1674f1e9-d0f3-49cf-9c20-40d72e9428e9\" (UID: \"1674f1e9-d0f3-49cf-9c20-40d72e9428e9\") " Apr 16 14:01:51.429195 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.429162 2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-alertmanager-trusted-ca-bundle\") pod \"1674f1e9-d0f3-49cf-9c20-40d72e9428e9\" (UID: \"1674f1e9-d0f3-49cf-9c20-40d72e9428e9\") " Apr 16 14:01:51.429426 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.429205 2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-cluster-tls-config\") pod \"1674f1e9-d0f3-49cf-9c20-40d72e9428e9\" (UID: \"1674f1e9-d0f3-49cf-9c20-40d72e9428e9\") " Apr 16 14:01:51.429426 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.429246 2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-secret-alertmanager-main-tls\") pod \"1674f1e9-d0f3-49cf-9c20-40d72e9428e9\" (UID: \"1674f1e9-d0f3-49cf-9c20-40d72e9428e9\") " Apr 16 14:01:51.429426 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.429277 2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-web-config\") pod \"1674f1e9-d0f3-49cf-9c20-40d72e9428e9\" (UID: \"1674f1e9-d0f3-49cf-9c20-40d72e9428e9\") " Apr 16 14:01:51.429426 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.429312 2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"1674f1e9-d0f3-49cf-9c20-40d72e9428e9\" (UID: \"1674f1e9-d0f3-49cf-9c20-40d72e9428e9\") " Apr 16 14:01:51.429426 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.429336 2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-tls-assets\") pod \"1674f1e9-d0f3-49cf-9c20-40d72e9428e9\" (UID: \"1674f1e9-d0f3-49cf-9c20-40d72e9428e9\") " Apr 16 14:01:51.429426 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.429373 2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-alertmanager-main-db\") pod \"1674f1e9-d0f3-49cf-9c20-40d72e9428e9\" (UID: \"1674f1e9-d0f3-49cf-9c20-40d72e9428e9\") " Apr 16 14:01:51.429426 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.429419 2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-config-volume\") pod \"1674f1e9-d0f3-49cf-9c20-40d72e9428e9\" (UID: \"1674f1e9-d0f3-49cf-9c20-40d72e9428e9\") " Apr 16 14:01:51.429799 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.429465 2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-secret-alertmanager-kube-rbac-proxy\") pod \"1674f1e9-d0f3-49cf-9c20-40d72e9428e9\" (UID: \"1674f1e9-d0f3-49cf-9c20-40d72e9428e9\") " Apr 16 14:01:51.429799 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.429503 2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-config-out\") pod \"1674f1e9-d0f3-49cf-9c20-40d72e9428e9\" (UID: \"1674f1e9-d0f3-49cf-9c20-40d72e9428e9\") " Apr 16 14:01:51.429799 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.429530 2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-secret-alertmanager-kube-rbac-proxy-web\") pod \"1674f1e9-d0f3-49cf-9c20-40d72e9428e9\" (UID: \"1674f1e9-d0f3-49cf-9c20-40d72e9428e9\") " Apr 16 14:01:51.429799 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.429576 2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-metrics-client-ca\") pod \"1674f1e9-d0f3-49cf-9c20-40d72e9428e9\" (UID: \"1674f1e9-d0f3-49cf-9c20-40d72e9428e9\") " Apr 16 14:01:51.429799 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.429603 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "1674f1e9-d0f3-49cf-9c20-40d72e9428e9" (UID: "1674f1e9-d0f3-49cf-9c20-40d72e9428e9"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 16 14:01:51.430057 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.429885 2574 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-136-114.ec2.internal\" DevicePath \"\"" Apr 16 14:01:51.430842 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.430489 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "1674f1e9-d0f3-49cf-9c20-40d72e9428e9" (UID: "1674f1e9-d0f3-49cf-9c20-40d72e9428e9"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 16 14:01:51.431138 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.431109 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "1674f1e9-d0f3-49cf-9c20-40d72e9428e9" (UID: "1674f1e9-d0f3-49cf-9c20-40d72e9428e9"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 16 14:01:51.432092 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.432050 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-kube-api-access-8sbs9" (OuterVolumeSpecName: "kube-api-access-8sbs9") pod "1674f1e9-d0f3-49cf-9c20-40d72e9428e9" (UID: "1674f1e9-d0f3-49cf-9c20-40d72e9428e9"). InnerVolumeSpecName "kube-api-access-8sbs9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 16 14:01:51.432298 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.432272 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "1674f1e9-d0f3-49cf-9c20-40d72e9428e9" (UID: "1674f1e9-d0f3-49cf-9c20-40d72e9428e9"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 16 14:01:51.432723 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.432689 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "1674f1e9-d0f3-49cf-9c20-40d72e9428e9" (UID: "1674f1e9-d0f3-49cf-9c20-40d72e9428e9"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 16 14:01:51.432920 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.432900 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "1674f1e9-d0f3-49cf-9c20-40d72e9428e9" (UID: "1674f1e9-d0f3-49cf-9c20-40d72e9428e9"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 16 14:01:51.433214 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.433194 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-config-out" (OuterVolumeSpecName: "config-out") pod "1674f1e9-d0f3-49cf-9c20-40d72e9428e9" (UID: "1674f1e9-d0f3-49cf-9c20-40d72e9428e9"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 16 14:01:51.433426 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.433410 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "1674f1e9-d0f3-49cf-9c20-40d72e9428e9" (UID: "1674f1e9-d0f3-49cf-9c20-40d72e9428e9"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 16 14:01:51.433795 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.433778 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-config-volume" (OuterVolumeSpecName: "config-volume") pod "1674f1e9-d0f3-49cf-9c20-40d72e9428e9" (UID: "1674f1e9-d0f3-49cf-9c20-40d72e9428e9"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 16 14:01:51.434292 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.434264 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "1674f1e9-d0f3-49cf-9c20-40d72e9428e9" (UID: "1674f1e9-d0f3-49cf-9c20-40d72e9428e9"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 16 14:01:51.436623 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.436596 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "1674f1e9-d0f3-49cf-9c20-40d72e9428e9" (UID: "1674f1e9-d0f3-49cf-9c20-40d72e9428e9"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 16 14:01:51.442978 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.442958 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-web-config" (OuterVolumeSpecName: "web-config") pod "1674f1e9-d0f3-49cf-9c20-40d72e9428e9" (UID: "1674f1e9-d0f3-49cf-9c20-40d72e9428e9"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 16 14:01:51.530856 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.530812 2574 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-alertmanager-main-db\") on node \"ip-10-0-136-114.ec2.internal\" DevicePath \"\"" Apr 16 14:01:51.530856 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.530852 2574 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-config-volume\") on node \"ip-10-0-136-114.ec2.internal\" DevicePath \"\"" Apr 16 14:01:51.530856 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.530864 2574 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-136-114.ec2.internal\" DevicePath \"\"" Apr 16 14:01:51.530856 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.530874 2574 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-config-out\") on node \"ip-10-0-136-114.ec2.internal\" DevicePath \"\"" Apr 16 14:01:51.531086 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.530883 2574 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-136-114.ec2.internal\" DevicePath \"\"" Apr 16 14:01:51.531086 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.530892 2574 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-metrics-client-ca\") on node \"ip-10-0-136-114.ec2.internal\" DevicePath \"\"" Apr 16 14:01:51.531086 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.530901 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sbs9\" (UniqueName: \"kubernetes.io/projected/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-kube-api-access-8sbs9\") on node \"ip-10-0-136-114.ec2.internal\" DevicePath \"\"" Apr 16 14:01:51.531086 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.530909 2574 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-cluster-tls-config\") on node \"ip-10-0-136-114.ec2.internal\" DevicePath \"\"" Apr 16 14:01:51.531086 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.530919 2574 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-secret-alertmanager-main-tls\") on node \"ip-10-0-136-114.ec2.internal\" DevicePath \"\"" Apr 16 14:01:51.531086 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.530928 2574 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-web-config\") on node \"ip-10-0-136-114.ec2.internal\" DevicePath \"\"" Apr 16 14:01:51.531086 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.530937 2574 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-136-114.ec2.internal\" DevicePath \"\"" Apr 16 14:01:51.531086 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.530945 2574 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1674f1e9-d0f3-49cf-9c20-40d72e9428e9-tls-assets\") on node \"ip-10-0-136-114.ec2.internal\" DevicePath \"\"" Apr 16 14:01:51.582966 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.582933 2574 generic.go:358] "Generic (PLEG): container finished" podID="1674f1e9-d0f3-49cf-9c20-40d72e9428e9" containerID="9c1100377039bff38551f96b0a7bb53031da5e82e496d8d6f7e56e48d24a9c89" exitCode=0 Apr 16 14:01:51.583119 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.583008 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1674f1e9-d0f3-49cf-9c20-40d72e9428e9","Type":"ContainerDied","Data":"9c1100377039bff38551f96b0a7bb53031da5e82e496d8d6f7e56e48d24a9c89"} Apr 16 14:01:51.583119 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.583045 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1674f1e9-d0f3-49cf-9c20-40d72e9428e9","Type":"ContainerDied","Data":"3e07d94cd968c488d5f1c5c3bf270784acc007d3e3e0c57e51133a06d7b0b0c4"} Apr 16 14:01:51.583119 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.583061 2574 scope.go:117] "RemoveContainer" containerID="cc125dfb019a0fc9f2aa8b910005d58e61f61ca47f96df371db6bd86f9c08d0f" Apr 16 14:01:51.583119 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.583069 2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:01:51.590167 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.590151 2574 scope.go:117] "RemoveContainer" containerID="57ef3cdb8657c272c6327739170ad1fc5a64a07e80709e961b8c458ec9cbbe51" Apr 16 14:01:51.596687 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.596672 2574 scope.go:117] "RemoveContainer" containerID="8a107c531e665503ab11c11b3c3393f93b4baf07bed584f9d77e9b0b398d2400" Apr 16 14:01:51.602756 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.602722 2574 scope.go:117] "RemoveContainer" containerID="9c1100377039bff38551f96b0a7bb53031da5e82e496d8d6f7e56e48d24a9c89" Apr 16 14:01:51.606710 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.606688 2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Apr 16 14:01:51.609510 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.609490 2574 scope.go:117] "RemoveContainer" containerID="2a98591981bae2c95137ab5cb7fa07f6e15c6fd690e1d01717204617f5248513" Apr 16 14:01:51.613098 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.613075 2574 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Apr 16 14:01:51.618145 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.618125 2574 scope.go:117] "RemoveContainer" containerID="c98b6e56106018578f36bcfd1e52ce2f0a937c05220ba15225e72d68577b5675" Apr 16 14:01:51.624628 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.624612 2574 scope.go:117] "RemoveContainer" containerID="fc8488bc4c0ac26656b5829766f499e968a418eda525d0881b36c8ae0d3881de" Apr 16 14:01:51.630679 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.630647 2574 scope.go:117] "RemoveContainer" containerID="cc125dfb019a0fc9f2aa8b910005d58e61f61ca47f96df371db6bd86f9c08d0f" Apr 16 14:01:51.630943 ip-10-0-136-114 kubenswrapper[2574]: E0416 14:01:51.630924 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc125dfb019a0fc9f2aa8b910005d58e61f61ca47f96df371db6bd86f9c08d0f\": container with ID starting with cc125dfb019a0fc9f2aa8b910005d58e61f61ca47f96df371db6bd86f9c08d0f not found: ID does not exist" containerID="cc125dfb019a0fc9f2aa8b910005d58e61f61ca47f96df371db6bd86f9c08d0f" Apr 16 14:01:51.631064 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.630959 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc125dfb019a0fc9f2aa8b910005d58e61f61ca47f96df371db6bd86f9c08d0f"} err="failed to get container status \"cc125dfb019a0fc9f2aa8b910005d58e61f61ca47f96df371db6bd86f9c08d0f\": rpc error: code = NotFound desc = could not find container \"cc125dfb019a0fc9f2aa8b910005d58e61f61ca47f96df371db6bd86f9c08d0f\": container with ID starting with cc125dfb019a0fc9f2aa8b910005d58e61f61ca47f96df371db6bd86f9c08d0f not found: ID does not exist" Apr 16 14:01:51.631064 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.630978 2574 scope.go:117] "RemoveContainer" containerID="57ef3cdb8657c272c6327739170ad1fc5a64a07e80709e961b8c458ec9cbbe51" Apr 16 14:01:51.631211 ip-10-0-136-114 kubenswrapper[2574]: E0416 14:01:51.631181 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57ef3cdb8657c272c6327739170ad1fc5a64a07e80709e961b8c458ec9cbbe51\": container with ID starting with 57ef3cdb8657c272c6327739170ad1fc5a64a07e80709e961b8c458ec9cbbe51 not found: ID does not exist" containerID="57ef3cdb8657c272c6327739170ad1fc5a64a07e80709e961b8c458ec9cbbe51" Apr 16 14:01:51.631249 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.631223 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57ef3cdb8657c272c6327739170ad1fc5a64a07e80709e961b8c458ec9cbbe51"} err="failed to get container status \"57ef3cdb8657c272c6327739170ad1fc5a64a07e80709e961b8c458ec9cbbe51\": rpc error: code = NotFound desc = could not find container \"57ef3cdb8657c272c6327739170ad1fc5a64a07e80709e961b8c458ec9cbbe51\": container with ID starting with 57ef3cdb8657c272c6327739170ad1fc5a64a07e80709e961b8c458ec9cbbe51 not found: ID does not exist" Apr 16 14:01:51.631249 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.631245 2574 scope.go:117] "RemoveContainer" containerID="8a107c531e665503ab11c11b3c3393f93b4baf07bed584f9d77e9b0b398d2400" Apr 16 14:01:51.631462 ip-10-0-136-114 kubenswrapper[2574]: E0416 14:01:51.631445 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a107c531e665503ab11c11b3c3393f93b4baf07bed584f9d77e9b0b398d2400\": container with ID starting with 8a107c531e665503ab11c11b3c3393f93b4baf07bed584f9d77e9b0b398d2400 not found: ID does not exist" containerID="8a107c531e665503ab11c11b3c3393f93b4baf07bed584f9d77e9b0b398d2400" Apr 16 14:01:51.631502 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.631466 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a107c531e665503ab11c11b3c3393f93b4baf07bed584f9d77e9b0b398d2400"} err="failed to get container status \"8a107c531e665503ab11c11b3c3393f93b4baf07bed584f9d77e9b0b398d2400\": rpc error: code = NotFound desc = could not find container \"8a107c531e665503ab11c11b3c3393f93b4baf07bed584f9d77e9b0b398d2400\": container with ID starting with 8a107c531e665503ab11c11b3c3393f93b4baf07bed584f9d77e9b0b398d2400 not found: ID does not exist" Apr 16 14:01:51.631502 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.631479 2574 scope.go:117] "RemoveContainer" containerID="9c1100377039bff38551f96b0a7bb53031da5e82e496d8d6f7e56e48d24a9c89" Apr 16 14:01:51.631708 ip-10-0-136-114 kubenswrapper[2574]: E0416 14:01:51.631689 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c1100377039bff38551f96b0a7bb53031da5e82e496d8d6f7e56e48d24a9c89\": container with ID starting with 9c1100377039bff38551f96b0a7bb53031da5e82e496d8d6f7e56e48d24a9c89 not found: ID does not exist" containerID="9c1100377039bff38551f96b0a7bb53031da5e82e496d8d6f7e56e48d24a9c89" Apr 16 14:01:51.631791 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.631716 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c1100377039bff38551f96b0a7bb53031da5e82e496d8d6f7e56e48d24a9c89"} err="failed to get container status \"9c1100377039bff38551f96b0a7bb53031da5e82e496d8d6f7e56e48d24a9c89\": rpc error: code = NotFound desc = could not find container \"9c1100377039bff38551f96b0a7bb53031da5e82e496d8d6f7e56e48d24a9c89\": container with ID starting with 9c1100377039bff38551f96b0a7bb53031da5e82e496d8d6f7e56e48d24a9c89 not found: ID does not exist" Apr 16 14:01:51.631791 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.631774 2574 scope.go:117] "RemoveContainer" containerID="2a98591981bae2c95137ab5cb7fa07f6e15c6fd690e1d01717204617f5248513" Apr 16 14:01:51.632018 ip-10-0-136-114 kubenswrapper[2574]: E0416 14:01:51.632003 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a98591981bae2c95137ab5cb7fa07f6e15c6fd690e1d01717204617f5248513\": container with ID starting with 2a98591981bae2c95137ab5cb7fa07f6e15c6fd690e1d01717204617f5248513 not found: ID does not exist" containerID="2a98591981bae2c95137ab5cb7fa07f6e15c6fd690e1d01717204617f5248513" Apr 16 14:01:51.632061 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.632022 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a98591981bae2c95137ab5cb7fa07f6e15c6fd690e1d01717204617f5248513"} err="failed to get container status \"2a98591981bae2c95137ab5cb7fa07f6e15c6fd690e1d01717204617f5248513\": rpc error: code = NotFound desc = could not find container \"2a98591981bae2c95137ab5cb7fa07f6e15c6fd690e1d01717204617f5248513\": container with ID starting with 2a98591981bae2c95137ab5cb7fa07f6e15c6fd690e1d01717204617f5248513 not found: ID does not exist" Apr 16 14:01:51.632061 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.632039 2574 scope.go:117] "RemoveContainer" containerID="c98b6e56106018578f36bcfd1e52ce2f0a937c05220ba15225e72d68577b5675" Apr 16 14:01:51.632280 ip-10-0-136-114 kubenswrapper[2574]: E0416 14:01:51.632262 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c98b6e56106018578f36bcfd1e52ce2f0a937c05220ba15225e72d68577b5675\": container with ID starting with c98b6e56106018578f36bcfd1e52ce2f0a937c05220ba15225e72d68577b5675 not found: ID does not exist" containerID="c98b6e56106018578f36bcfd1e52ce2f0a937c05220ba15225e72d68577b5675" Apr 16 14:01:51.632330 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.632285 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c98b6e56106018578f36bcfd1e52ce2f0a937c05220ba15225e72d68577b5675"} err="failed to get container status \"c98b6e56106018578f36bcfd1e52ce2f0a937c05220ba15225e72d68577b5675\": rpc error: code = NotFound desc = could not find container \"c98b6e56106018578f36bcfd1e52ce2f0a937c05220ba15225e72d68577b5675\": container with ID starting with c98b6e56106018578f36bcfd1e52ce2f0a937c05220ba15225e72d68577b5675 not found: ID does not exist" Apr 16 14:01:51.632330 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.632302 2574 scope.go:117] "RemoveContainer" containerID="fc8488bc4c0ac26656b5829766f499e968a418eda525d0881b36c8ae0d3881de" Apr 16 14:01:51.632528 ip-10-0-136-114 kubenswrapper[2574]: E0416 14:01:51.632513 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc8488bc4c0ac26656b5829766f499e968a418eda525d0881b36c8ae0d3881de\": container with ID starting with fc8488bc4c0ac26656b5829766f499e968a418eda525d0881b36c8ae0d3881de not found: ID does not exist" containerID="fc8488bc4c0ac26656b5829766f499e968a418eda525d0881b36c8ae0d3881de" Apr 16 14:01:51.632576 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.632531 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc8488bc4c0ac26656b5829766f499e968a418eda525d0881b36c8ae0d3881de"} err="failed to get container status \"fc8488bc4c0ac26656b5829766f499e968a418eda525d0881b36c8ae0d3881de\": rpc error: code = NotFound desc = could not find container \"fc8488bc4c0ac26656b5829766f499e968a418eda525d0881b36c8ae0d3881de\": container with ID starting with fc8488bc4c0ac26656b5829766f499e968a418eda525d0881b36c8ae0d3881de not found: ID does not exist" Apr 16 14:01:51.639940 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.639920 2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Apr 16 14:01:51.640319 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.640303 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1674f1e9-d0f3-49cf-9c20-40d72e9428e9" containerName="prom-label-proxy" Apr 16 14:01:51.640407 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.640322 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1674f1e9-d0f3-49cf-9c20-40d72e9428e9" containerName="prom-label-proxy" Apr 16 14:01:51.640407 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.640335 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1674f1e9-d0f3-49cf-9c20-40d72e9428e9" containerName="kube-rbac-proxy-metric" Apr 16 14:01:51.640407 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.640343 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1674f1e9-d0f3-49cf-9c20-40d72e9428e9" containerName="kube-rbac-proxy-metric" Apr 16 14:01:51.640407 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.640358 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee5dbe45-5126-4c3e-9397-69e2153637de" containerName="console" Apr 16 14:01:51.640407 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.640369 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee5dbe45-5126-4c3e-9397-69e2153637de" containerName="console" Apr 16 14:01:51.640407 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.640381 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1674f1e9-d0f3-49cf-9c20-40d72e9428e9" containerName="config-reloader" Apr 16 14:01:51.640407 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.640389 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1674f1e9-d0f3-49cf-9c20-40d72e9428e9" containerName="config-reloader" Apr 16 14:01:51.640407 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.640400 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1674f1e9-d0f3-49cf-9c20-40d72e9428e9" containerName="kube-rbac-proxy" Apr 16 14:01:51.640815 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.640408 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1674f1e9-d0f3-49cf-9c20-40d72e9428e9" containerName="kube-rbac-proxy" Apr 16 14:01:51.640815 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.640515 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1674f1e9-d0f3-49cf-9c20-40d72e9428e9" containerName="init-config-reloader" Apr 16 14:01:51.640815 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.640524 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1674f1e9-d0f3-49cf-9c20-40d72e9428e9" containerName="init-config-reloader" Apr 16 14:01:51.640815 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.640540 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1674f1e9-d0f3-49cf-9c20-40d72e9428e9" containerName="alertmanager" Apr 16 14:01:51.640815 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.640549 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1674f1e9-d0f3-49cf-9c20-40d72e9428e9" containerName="alertmanager" Apr 16 14:01:51.640815 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.640559 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1674f1e9-d0f3-49cf-9c20-40d72e9428e9" containerName="kube-rbac-proxy-web" Apr 16 14:01:51.640815 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.640567 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1674f1e9-d0f3-49cf-9c20-40d72e9428e9" containerName="kube-rbac-proxy-web" Apr 16 14:01:51.641134 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.640822 2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="1674f1e9-d0f3-49cf-9c20-40d72e9428e9" containerName="kube-rbac-proxy-metric" Apr 16 14:01:51.641134 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.640873 2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="1674f1e9-d0f3-49cf-9c20-40d72e9428e9" containerName="prom-label-proxy" Apr 16 14:01:51.641134 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.640886 2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="1674f1e9-d0f3-49cf-9c20-40d72e9428e9" containerName="kube-rbac-proxy-web" Apr 16 14:01:51.641134 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.640907 2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="1674f1e9-d0f3-49cf-9c20-40d72e9428e9" containerName="kube-rbac-proxy" Apr 16 14:01:51.641134 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.640919 2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="1674f1e9-d0f3-49cf-9c20-40d72e9428e9" containerName="alertmanager" Apr 16 14:01:51.641134 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.640933 2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="1674f1e9-d0f3-49cf-9c20-40d72e9428e9" containerName="config-reloader" Apr 16 14:01:51.641134 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.640957 2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="ee5dbe45-5126-4c3e-9397-69e2153637de" containerName="console" Apr 16 14:01:51.644163 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.644142 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:01:51.646522 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.646503 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Apr 16 14:01:51.646905 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.646768 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Apr 16 14:01:51.646905 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.646768 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Apr 16 14:01:51.646905 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.646851 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Apr 16 14:01:51.646905 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.646881 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Apr 16 14:01:51.647154 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.646883 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Apr 16 14:01:51.647154 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.646960 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Apr 16 14:01:51.647154 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.646794 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Apr 16 14:01:51.647616 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.647597 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-v2qjl\"" Apr 16 14:01:51.651592 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.651575 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Apr 16 14:01:51.655329 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.655309 2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Apr 16 14:01:51.732535 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.732463 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/59f76eb3-ff88-4e98-9015-4a91c304f5ec-tls-assets\") pod \"alertmanager-main-0\" (UID: \"59f76eb3-ff88-4e98-9015-4a91c304f5ec\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:01:51.732535 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.732512 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/59f76eb3-ff88-4e98-9015-4a91c304f5ec-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"59f76eb3-ff88-4e98-9015-4a91c304f5ec\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:01:51.732677 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.732535 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/59f76eb3-ff88-4e98-9015-4a91c304f5ec-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"59f76eb3-ff88-4e98-9015-4a91c304f5ec\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:01:51.732677 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.732609 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/59f76eb3-ff88-4e98-9015-4a91c304f5ec-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"59f76eb3-ff88-4e98-9015-4a91c304f5ec\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:01:51.732677 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.732661 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/59f76eb3-ff88-4e98-9015-4a91c304f5ec-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"59f76eb3-ff88-4e98-9015-4a91c304f5ec\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:01:51.732796 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.732694 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/59f76eb3-ff88-4e98-9015-4a91c304f5ec-config-out\") pod \"alertmanager-main-0\" (UID: \"59f76eb3-ff88-4e98-9015-4a91c304f5ec\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:01:51.732796 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.732715 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdt9j\" (UniqueName: \"kubernetes.io/projected/59f76eb3-ff88-4e98-9015-4a91c304f5ec-kube-api-access-tdt9j\") pod \"alertmanager-main-0\" (UID: \"59f76eb3-ff88-4e98-9015-4a91c304f5ec\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:01:51.732796 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.732762 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/59f76eb3-ff88-4e98-9015-4a91c304f5ec-config-volume\") pod \"alertmanager-main-0\" (UID: \"59f76eb3-ff88-4e98-9015-4a91c304f5ec\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:01:51.732796 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.732783 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/59f76eb3-ff88-4e98-9015-4a91c304f5ec-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"59f76eb3-ff88-4e98-9015-4a91c304f5ec\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:01:51.732912 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.732809 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/59f76eb3-ff88-4e98-9015-4a91c304f5ec-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"59f76eb3-ff88-4e98-9015-4a91c304f5ec\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:01:51.732912 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.732825 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/59f76eb3-ff88-4e98-9015-4a91c304f5ec-web-config\") pod \"alertmanager-main-0\" (UID: \"59f76eb3-ff88-4e98-9015-4a91c304f5ec\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:01:51.732912 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.732851 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59f76eb3-ff88-4e98-9015-4a91c304f5ec-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"59f76eb3-ff88-4e98-9015-4a91c304f5ec\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:01:51.732912 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.732880 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/59f76eb3-ff88-4e98-9015-4a91c304f5ec-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"59f76eb3-ff88-4e98-9015-4a91c304f5ec\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:01:51.834238 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.834199 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tdt9j\" (UniqueName: \"kubernetes.io/projected/59f76eb3-ff88-4e98-9015-4a91c304f5ec-kube-api-access-tdt9j\") pod \"alertmanager-main-0\" (UID: \"59f76eb3-ff88-4e98-9015-4a91c304f5ec\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:01:51.834402 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.834250 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/59f76eb3-ff88-4e98-9015-4a91c304f5ec-config-volume\") pod \"alertmanager-main-0\" (UID: \"59f76eb3-ff88-4e98-9015-4a91c304f5ec\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:01:51.834402 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.834362 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/59f76eb3-ff88-4e98-9015-4a91c304f5ec-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"59f76eb3-ff88-4e98-9015-4a91c304f5ec\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:01:51.834473 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.834413 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/59f76eb3-ff88-4e98-9015-4a91c304f5ec-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"59f76eb3-ff88-4e98-9015-4a91c304f5ec\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:01:51.834473 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.834432 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/59f76eb3-ff88-4e98-9015-4a91c304f5ec-web-config\") pod \"alertmanager-main-0\" (UID: \"59f76eb3-ff88-4e98-9015-4a91c304f5ec\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:01:51.834473 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.834450 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59f76eb3-ff88-4e98-9015-4a91c304f5ec-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"59f76eb3-ff88-4e98-9015-4a91c304f5ec\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:01:51.834610 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.834478 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/59f76eb3-ff88-4e98-9015-4a91c304f5ec-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"59f76eb3-ff88-4e98-9015-4a91c304f5ec\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:01:51.834677 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.834654 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/59f76eb3-ff88-4e98-9015-4a91c304f5ec-tls-assets\") pod \"alertmanager-main-0\" (UID: \"59f76eb3-ff88-4e98-9015-4a91c304f5ec\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:01:51.834955 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.834927 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/59f76eb3-ff88-4e98-9015-4a91c304f5ec-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"59f76eb3-ff88-4e98-9015-4a91c304f5ec\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:01:51.835050 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.834975 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/59f76eb3-ff88-4e98-9015-4a91c304f5ec-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"59f76eb3-ff88-4e98-9015-4a91c304f5ec\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:01:51.835050 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.835028 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/59f76eb3-ff88-4e98-9015-4a91c304f5ec-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"59f76eb3-ff88-4e98-9015-4a91c304f5ec\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:01:51.835158 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.835075 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/59f76eb3-ff88-4e98-9015-4a91c304f5ec-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"59f76eb3-ff88-4e98-9015-4a91c304f5ec\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:01:51.835438 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.835409 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/59f76eb3-ff88-4e98-9015-4a91c304f5ec-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"59f76eb3-ff88-4e98-9015-4a91c304f5ec\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:01:51.835791 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.835767 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/59f76eb3-ff88-4e98-9015-4a91c304f5ec-config-out\") pod \"alertmanager-main-0\" (UID: \"59f76eb3-ff88-4e98-9015-4a91c304f5ec\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:01:51.836085 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.836058 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/59f76eb3-ff88-4e98-9015-4a91c304f5ec-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"59f76eb3-ff88-4e98-9015-4a91c304f5ec\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:01:51.836224 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.836203 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59f76eb3-ff88-4e98-9015-4a91c304f5ec-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"59f76eb3-ff88-4e98-9015-4a91c304f5ec\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:01:51.837609 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.837560 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/59f76eb3-ff88-4e98-9015-4a91c304f5ec-tls-assets\") pod \"alertmanager-main-0\" (UID: \"59f76eb3-ff88-4e98-9015-4a91c304f5ec\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:01:51.837609 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.837577 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/59f76eb3-ff88-4e98-9015-4a91c304f5ec-config-volume\") pod \"alertmanager-main-0\" (UID: \"59f76eb3-ff88-4e98-9015-4a91c304f5ec\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:01:51.837782 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.837654 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/59f76eb3-ff88-4e98-9015-4a91c304f5ec-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"59f76eb3-ff88-4e98-9015-4a91c304f5ec\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:01:51.838040 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.838016 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/59f76eb3-ff88-4e98-9015-4a91c304f5ec-web-config\") pod \"alertmanager-main-0\" (UID: \"59f76eb3-ff88-4e98-9015-4a91c304f5ec\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:01:51.838126 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.838013 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/59f76eb3-ff88-4e98-9015-4a91c304f5ec-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"59f76eb3-ff88-4e98-9015-4a91c304f5ec\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:01:51.838490 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.838468 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/59f76eb3-ff88-4e98-9015-4a91c304f5ec-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"59f76eb3-ff88-4e98-9015-4a91c304f5ec\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:01:51.838545 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.838530 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/59f76eb3-ff88-4e98-9015-4a91c304f5ec-config-out\") pod \"alertmanager-main-0\" (UID: \"59f76eb3-ff88-4e98-9015-4a91c304f5ec\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:01:51.839012 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.838997 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/59f76eb3-ff88-4e98-9015-4a91c304f5ec-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"59f76eb3-ff88-4e98-9015-4a91c304f5ec\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:01:51.839057 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.839034 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/59f76eb3-ff88-4e98-9015-4a91c304f5ec-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"59f76eb3-ff88-4e98-9015-4a91c304f5ec\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:01:51.845673 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.845652 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdt9j\" (UniqueName: \"kubernetes.io/projected/59f76eb3-ff88-4e98-9015-4a91c304f5ec-kube-api-access-tdt9j\") pod \"alertmanager-main-0\" (UID: \"59f76eb3-ff88-4e98-9015-4a91c304f5ec\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:01:51.954286 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:51.954254 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Apr 16 14:01:52.080464 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:52.080412 2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Apr 16 14:01:52.084536 ip-10-0-136-114 kubenswrapper[2574]: W0416 14:01:52.084498 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59f76eb3_ff88_4e98_9015_4a91c304f5ec.slice/crio-bdb00b16507aa95a6efe14dab4a864e96fad766368f64b874da037f0b6cee0d5 WatchSource:0}: Error finding container bdb00b16507aa95a6efe14dab4a864e96fad766368f64b874da037f0b6cee0d5: Status 404 returned error can't find the container with id bdb00b16507aa95a6efe14dab4a864e96fad766368f64b874da037f0b6cee0d5 Apr 16 14:01:52.174446 ip-10-0-136-114 kubenswrapper[2574]: E0416 14:01:52.174404 2574 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-lnfdm" podUID="eded8e93-5cbc-435c-aab3-a909812ba63b" Apr 16 14:01:52.588209 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:52.588175 2574 generic.go:358] "Generic (PLEG): container finished" podID="59f76eb3-ff88-4e98-9015-4a91c304f5ec" containerID="1e1acd659c4e4605293a7d11f5acc33a2211a45192e9ec85dcbf07223e11fe73" exitCode=0 Apr 16 14:01:52.588688 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:52.588220 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"59f76eb3-ff88-4e98-9015-4a91c304f5ec","Type":"ContainerDied","Data":"1e1acd659c4e4605293a7d11f5acc33a2211a45192e9ec85dcbf07223e11fe73"} Apr 16 14:01:52.588688 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:52.588260 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"59f76eb3-ff88-4e98-9015-4a91c304f5ec","Type":"ContainerStarted","Data":"bdb00b16507aa95a6efe14dab4a864e96fad766368f64b874da037f0b6cee0d5"} Apr 16 14:01:52.588688 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:52.588278 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lnfdm" Apr 16 14:01:52.811996 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:52.811968 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1674f1e9-d0f3-49cf-9c20-40d72e9428e9" path="/var/lib/kubelet/pods/1674f1e9-d0f3-49cf-9c20-40d72e9428e9/volumes" Apr 16 14:01:53.594655 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:53.594621 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"59f76eb3-ff88-4e98-9015-4a91c304f5ec","Type":"ContainerStarted","Data":"28851e79a8427bffa405a7313b67911d8e0655226701082a4ed7232dd3fd9201"} Apr 16 14:01:53.594655 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:53.594659 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"59f76eb3-ff88-4e98-9015-4a91c304f5ec","Type":"ContainerStarted","Data":"959aaf242f55617e1fb2ec68b86098a16d5ec3cf3cdc3713a797dc2742bcb62d"} Apr 16 14:01:53.595129 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:53.594675 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"59f76eb3-ff88-4e98-9015-4a91c304f5ec","Type":"ContainerStarted","Data":"a7290f2672884e7769aad458ba3e2127f6608d6ca6a579349bb7fe8f04a58d2f"} Apr 16 14:01:53.595129 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:53.594686 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"59f76eb3-ff88-4e98-9015-4a91c304f5ec","Type":"ContainerStarted","Data":"a58eab4e2f79ac2859a145517f216bcbafed51b2a3166f63136e79fc00d8b73e"} Apr 16 14:01:53.595129 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:53.594697 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"59f76eb3-ff88-4e98-9015-4a91c304f5ec","Type":"ContainerStarted","Data":"f7f0fe2de095ca6e08a1b9bd8103355e0dacb4aa78faf28da889203432a0540e"} Apr 16 14:01:53.595129 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:53.594709 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"59f76eb3-ff88-4e98-9015-4a91c304f5ec","Type":"ContainerStarted","Data":"9035bda8c18a0bb061e8432b3072d60d53a25c47d4fb4fe959931bd686764fee"} Apr 16 14:01:53.621601 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:53.621554 2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.6215412369999997 podStartE2EDuration="2.621541237s" podCreationTimestamp="2026-04-16 14:01:51 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 14:01:53.62021524 +0000 UTC m=+283.449950110" watchObservedRunningTime="2026-04-16 14:01:53.621541237 +0000 UTC m=+283.451276108" Apr 16 14:01:54.108439 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:54.108406 2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-64476d7dbf-cmlpq"] Apr 16 14:01:54.110906 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:54.110890 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-64476d7dbf-cmlpq" Apr 16 14:01:54.113773 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:54.113734 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Apr 16 14:01:54.113885 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:54.113792 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Apr 16 14:01:54.113885 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:54.113798 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Apr 16 14:01:54.113986 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:54.113823 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Apr 16 14:01:54.114089 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:54.114074 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Apr 16 14:01:54.114183 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:54.114166 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-tqn9d\"" Apr 16 14:01:54.120690 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:54.120668 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Apr 16 14:01:54.127890 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:54.127870 2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-64476d7dbf-cmlpq"] Apr 16 14:01:54.256409 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:54.256380 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/dacf88d2-d40d-4130-8f88-e060c7b4835c-secret-telemeter-client\") pod \"telemeter-client-64476d7dbf-cmlpq\" (UID: \"dacf88d2-d40d-4130-8f88-e060c7b4835c\") " pod="openshift-monitoring/telemeter-client-64476d7dbf-cmlpq" Apr 16 14:01:54.256409 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:54.256412 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/dacf88d2-d40d-4130-8f88-e060c7b4835c-federate-client-tls\") pod \"telemeter-client-64476d7dbf-cmlpq\" (UID: \"dacf88d2-d40d-4130-8f88-e060c7b4835c\") " pod="openshift-monitoring/telemeter-client-64476d7dbf-cmlpq" Apr 16 14:01:54.256592 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:54.256438 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dacf88d2-d40d-4130-8f88-e060c7b4835c-serving-certs-ca-bundle\") pod \"telemeter-client-64476d7dbf-cmlpq\" (UID: \"dacf88d2-d40d-4130-8f88-e060c7b4835c\") " pod="openshift-monitoring/telemeter-client-64476d7dbf-cmlpq" Apr 16 14:01:54.256592 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:54.256530 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/dacf88d2-d40d-4130-8f88-e060c7b4835c-telemeter-client-tls\") pod \"telemeter-client-64476d7dbf-cmlpq\" (UID: \"dacf88d2-d40d-4130-8f88-e060c7b4835c\") " pod="openshift-monitoring/telemeter-client-64476d7dbf-cmlpq" Apr 16 14:01:54.256592 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:54.256561 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dacf88d2-d40d-4130-8f88-e060c7b4835c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-64476d7dbf-cmlpq\" (UID: \"dacf88d2-d40d-4130-8f88-e060c7b4835c\") " pod="openshift-monitoring/telemeter-client-64476d7dbf-cmlpq" Apr 16 14:01:54.256694 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:54.256614 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvgtd\" (UniqueName: \"kubernetes.io/projected/dacf88d2-d40d-4130-8f88-e060c7b4835c-kube-api-access-pvgtd\") pod \"telemeter-client-64476d7dbf-cmlpq\" (UID: \"dacf88d2-d40d-4130-8f88-e060c7b4835c\") " pod="openshift-monitoring/telemeter-client-64476d7dbf-cmlpq" Apr 16 14:01:54.256694 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:54.256646 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dacf88d2-d40d-4130-8f88-e060c7b4835c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-64476d7dbf-cmlpq\" (UID: \"dacf88d2-d40d-4130-8f88-e060c7b4835c\") " pod="openshift-monitoring/telemeter-client-64476d7dbf-cmlpq" Apr 16 14:01:54.256778 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:54.256734 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dacf88d2-d40d-4130-8f88-e060c7b4835c-metrics-client-ca\") pod \"telemeter-client-64476d7dbf-cmlpq\" (UID: \"dacf88d2-d40d-4130-8f88-e060c7b4835c\") " pod="openshift-monitoring/telemeter-client-64476d7dbf-cmlpq" Apr 16 14:01:54.357931 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:54.357901 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dacf88d2-d40d-4130-8f88-e060c7b4835c-metrics-client-ca\") pod \"telemeter-client-64476d7dbf-cmlpq\" (UID: \"dacf88d2-d40d-4130-8f88-e060c7b4835c\") " pod="openshift-monitoring/telemeter-client-64476d7dbf-cmlpq" Apr 16 14:01:54.358083 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:54.357944 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/dacf88d2-d40d-4130-8f88-e060c7b4835c-secret-telemeter-client\") pod \"telemeter-client-64476d7dbf-cmlpq\" (UID: \"dacf88d2-d40d-4130-8f88-e060c7b4835c\") " pod="openshift-monitoring/telemeter-client-64476d7dbf-cmlpq" Apr 16 14:01:54.358083 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:54.357969 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/dacf88d2-d40d-4130-8f88-e060c7b4835c-federate-client-tls\") pod \"telemeter-client-64476d7dbf-cmlpq\" (UID: \"dacf88d2-d40d-4130-8f88-e060c7b4835c\") " pod="openshift-monitoring/telemeter-client-64476d7dbf-cmlpq" Apr 16 14:01:54.358083 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:54.357997 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dacf88d2-d40d-4130-8f88-e060c7b4835c-serving-certs-ca-bundle\") pod \"telemeter-client-64476d7dbf-cmlpq\" (UID: \"dacf88d2-d40d-4130-8f88-e060c7b4835c\") " pod="openshift-monitoring/telemeter-client-64476d7dbf-cmlpq" Apr 16 14:01:54.358246 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:54.358182 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/dacf88d2-d40d-4130-8f88-e060c7b4835c-telemeter-client-tls\") pod \"telemeter-client-64476d7dbf-cmlpq\" (UID: \"dacf88d2-d40d-4130-8f88-e060c7b4835c\") " pod="openshift-monitoring/telemeter-client-64476d7dbf-cmlpq" Apr 16 14:01:54.358246 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:54.358233 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dacf88d2-d40d-4130-8f88-e060c7b4835c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-64476d7dbf-cmlpq\" (UID: \"dacf88d2-d40d-4130-8f88-e060c7b4835c\") " pod="openshift-monitoring/telemeter-client-64476d7dbf-cmlpq" Apr 16 14:01:54.358348 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:54.358262 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pvgtd\" (UniqueName: \"kubernetes.io/projected/dacf88d2-d40d-4130-8f88-e060c7b4835c-kube-api-access-pvgtd\") pod \"telemeter-client-64476d7dbf-cmlpq\" (UID: \"dacf88d2-d40d-4130-8f88-e060c7b4835c\") " pod="openshift-monitoring/telemeter-client-64476d7dbf-cmlpq" Apr 16 14:01:54.358348 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:54.358316 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dacf88d2-d40d-4130-8f88-e060c7b4835c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-64476d7dbf-cmlpq\" (UID: \"dacf88d2-d40d-4130-8f88-e060c7b4835c\") " pod="openshift-monitoring/telemeter-client-64476d7dbf-cmlpq" Apr 16 14:01:54.358771 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:54.358681 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dacf88d2-d40d-4130-8f88-e060c7b4835c-metrics-client-ca\") pod \"telemeter-client-64476d7dbf-cmlpq\" (UID: \"dacf88d2-d40d-4130-8f88-e060c7b4835c\") " pod="openshift-monitoring/telemeter-client-64476d7dbf-cmlpq" Apr 16 14:01:54.358771 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:54.358709 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dacf88d2-d40d-4130-8f88-e060c7b4835c-serving-certs-ca-bundle\") pod \"telemeter-client-64476d7dbf-cmlpq\" (UID: \"dacf88d2-d40d-4130-8f88-e060c7b4835c\") " pod="openshift-monitoring/telemeter-client-64476d7dbf-cmlpq" Apr 16 14:01:54.359326 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:54.359298 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dacf88d2-d40d-4130-8f88-e060c7b4835c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-64476d7dbf-cmlpq\" (UID: \"dacf88d2-d40d-4130-8f88-e060c7b4835c\") " pod="openshift-monitoring/telemeter-client-64476d7dbf-cmlpq" Apr 16 14:01:54.360645 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:54.360622 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/dacf88d2-d40d-4130-8f88-e060c7b4835c-secret-telemeter-client\") pod \"telemeter-client-64476d7dbf-cmlpq\" (UID: \"dacf88d2-d40d-4130-8f88-e060c7b4835c\") " pod="openshift-monitoring/telemeter-client-64476d7dbf-cmlpq" Apr 16 14:01:54.360765 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:54.360668 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dacf88d2-d40d-4130-8f88-e060c7b4835c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-64476d7dbf-cmlpq\" (UID: \"dacf88d2-d40d-4130-8f88-e060c7b4835c\") " pod="openshift-monitoring/telemeter-client-64476d7dbf-cmlpq" Apr 16 14:01:54.360765 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:54.360683 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/dacf88d2-d40d-4130-8f88-e060c7b4835c-federate-client-tls\") pod \"telemeter-client-64476d7dbf-cmlpq\" (UID: \"dacf88d2-d40d-4130-8f88-e060c7b4835c\") " pod="openshift-monitoring/telemeter-client-64476d7dbf-cmlpq" Apr 16 14:01:54.360886 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:54.360872 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/dacf88d2-d40d-4130-8f88-e060c7b4835c-telemeter-client-tls\") pod \"telemeter-client-64476d7dbf-cmlpq\" (UID: \"dacf88d2-d40d-4130-8f88-e060c7b4835c\") " pod="openshift-monitoring/telemeter-client-64476d7dbf-cmlpq" Apr 16 14:01:54.366468 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:54.366444 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvgtd\" (UniqueName: \"kubernetes.io/projected/dacf88d2-d40d-4130-8f88-e060c7b4835c-kube-api-access-pvgtd\") pod \"telemeter-client-64476d7dbf-cmlpq\" (UID: \"dacf88d2-d40d-4130-8f88-e060c7b4835c\") " pod="openshift-monitoring/telemeter-client-64476d7dbf-cmlpq" Apr 16 14:01:54.421451 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:54.421427 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-64476d7dbf-cmlpq" Apr 16 14:01:54.545992 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:54.545969 2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-64476d7dbf-cmlpq"] Apr 16 14:01:54.548111 ip-10-0-136-114 kubenswrapper[2574]: W0416 14:01:54.548081 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddacf88d2_d40d_4130_8f88_e060c7b4835c.slice/crio-29c0662e975ecd5921f092b90b5767bcb11c50a4efc735488a0eee81b04c7973 WatchSource:0}: Error finding container 29c0662e975ecd5921f092b90b5767bcb11c50a4efc735488a0eee81b04c7973: Status 404 returned error can't find the container with id 29c0662e975ecd5921f092b90b5767bcb11c50a4efc735488a0eee81b04c7973 Apr 16 14:01:54.604380 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:54.604348 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-64476d7dbf-cmlpq" event={"ID":"dacf88d2-d40d-4130-8f88-e060c7b4835c","Type":"ContainerStarted","Data":"29c0662e975ecd5921f092b90b5767bcb11c50a4efc735488a0eee81b04c7973"} Apr 16 14:01:55.566414 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:55.566384 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ee1c9f7-5758-4734-a7b3-ea621cc6df95-cert\") pod \"ingress-canary-xjpgz\" (UID: \"7ee1c9f7-5758-4734-a7b3-ea621cc6df95\") " pod="openshift-ingress-canary/ingress-canary-xjpgz" Apr 16 14:01:55.566414 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:55.566416 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eded8e93-5cbc-435c-aab3-a909812ba63b-metrics-tls\") pod \"dns-default-lnfdm\" (UID: \"eded8e93-5cbc-435c-aab3-a909812ba63b\") " pod="openshift-dns/dns-default-lnfdm" Apr 16 14:01:55.568699 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:55.568674 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eded8e93-5cbc-435c-aab3-a909812ba63b-metrics-tls\") pod \"dns-default-lnfdm\" (UID: \"eded8e93-5cbc-435c-aab3-a909812ba63b\") " pod="openshift-dns/dns-default-lnfdm" Apr 16 14:01:55.568833 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:55.568813 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ee1c9f7-5758-4734-a7b3-ea621cc6df95-cert\") pod \"ingress-canary-xjpgz\" (UID: \"7ee1c9f7-5758-4734-a7b3-ea621cc6df95\") " pod="openshift-ingress-canary/ingress-canary-xjpgz" Apr 16 14:01:55.591204 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:55.591174 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ftzrr\"" Apr 16 14:01:55.599076 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:55.599052 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lnfdm" Apr 16 14:01:55.708790 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:55.708764 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-mv45m\"" Apr 16 14:01:55.716814 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:55.716790 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xjpgz" Apr 16 14:01:55.733129 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:55.732569 2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lnfdm"] Apr 16 14:01:55.735225 ip-10-0-136-114 kubenswrapper[2574]: W0416 14:01:55.735187 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeded8e93_5cbc_435c_aab3_a909812ba63b.slice/crio-42fdcbd06a43a42031006cc32cb8ed0e7884a3f197a20919df4712c97b69e2c7 WatchSource:0}: Error finding container 42fdcbd06a43a42031006cc32cb8ed0e7884a3f197a20919df4712c97b69e2c7: Status 404 returned error can't find the container with id 42fdcbd06a43a42031006cc32cb8ed0e7884a3f197a20919df4712c97b69e2c7 Apr 16 14:01:55.857765 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:55.857727 2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xjpgz"] Apr 16 14:01:55.860034 ip-10-0-136-114 kubenswrapper[2574]: W0416 14:01:55.860004 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ee1c9f7_5758_4734_a7b3_ea621cc6df95.slice/crio-87a416d89b4c028f36b3c750ff5ae054709ceb9ebbebfffbd22e4215b1b39ef3 WatchSource:0}: Error finding container 87a416d89b4c028f36b3c750ff5ae054709ceb9ebbebfffbd22e4215b1b39ef3: Status 404 returned error can't find the container with id 87a416d89b4c028f36b3c750ff5ae054709ceb9ebbebfffbd22e4215b1b39ef3 Apr 16 14:01:56.611332 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:56.611291 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xjpgz" event={"ID":"7ee1c9f7-5758-4734-a7b3-ea621cc6df95","Type":"ContainerStarted","Data":"87a416d89b4c028f36b3c750ff5ae054709ceb9ebbebfffbd22e4215b1b39ef3"} Apr 16 14:01:56.612366 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:56.612328 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lnfdm" event={"ID":"eded8e93-5cbc-435c-aab3-a909812ba63b","Type":"ContainerStarted","Data":"42fdcbd06a43a42031006cc32cb8ed0e7884a3f197a20919df4712c97b69e2c7"} Apr 16 14:01:57.618309 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:57.618265 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-64476d7dbf-cmlpq" event={"ID":"dacf88d2-d40d-4130-8f88-e060c7b4835c","Type":"ContainerStarted","Data":"ec577bcc6fb6c57e2e290dc1d710a68ee781e6d4d0a1e985e2a27efe013cb1a5"} Apr 16 14:01:57.618309 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:57.618311 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-64476d7dbf-cmlpq" event={"ID":"dacf88d2-d40d-4130-8f88-e060c7b4835c","Type":"ContainerStarted","Data":"e3a90f11280ea019425c40c35d82899e4663aa06ced0f4e05bf90187c21697a7"} Apr 16 14:01:57.618722 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:57.618321 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-64476d7dbf-cmlpq" event={"ID":"dacf88d2-d40d-4130-8f88-e060c7b4835c","Type":"ContainerStarted","Data":"9b5ac6b4c891cf439a40440c04b53e47fb4ea04a1fee56c1c1bf012463285f87"} Apr 16 14:01:57.642619 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:57.642570 2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-64476d7dbf-cmlpq" podStartSLOduration=1.446234402 podStartE2EDuration="3.642555625s" podCreationTimestamp="2026-04-16 14:01:54 +0000 UTC" firstStartedPulling="2026-04-16 14:01:54.550192425 +0000 UTC m=+284.379927273" lastFinishedPulling="2026-04-16 14:01:56.746513644 +0000 UTC m=+286.576248496" observedRunningTime="2026-04-16 14:01:57.639687787 +0000 UTC m=+287.469422665" watchObservedRunningTime="2026-04-16 14:01:57.642555625 +0000 UTC m=+287.472290486" Apr 16 14:01:58.327309 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:58.327285 2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-9b4885c74-tq47h"] Apr 16 14:01:58.330563 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:58.330542 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9b4885c74-tq47h" Apr 16 14:01:58.343993 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:58.343635 2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9b4885c74-tq47h"] Apr 16 14:01:58.495063 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:58.495013 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7fb4d384-4658-48c6-a920-4d5b878793d6-oauth-serving-cert\") pod \"console-9b4885c74-tq47h\" (UID: \"7fb4d384-4658-48c6-a920-4d5b878793d6\") " pod="openshift-console/console-9b4885c74-tq47h" Apr 16 14:01:58.495063 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:58.495067 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x2dc\" (UniqueName: \"kubernetes.io/projected/7fb4d384-4658-48c6-a920-4d5b878793d6-kube-api-access-4x2dc\") pod \"console-9b4885c74-tq47h\" (UID: \"7fb4d384-4658-48c6-a920-4d5b878793d6\") " pod="openshift-console/console-9b4885c74-tq47h" Apr 16 14:01:58.495291 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:58.495103 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7fb4d384-4658-48c6-a920-4d5b878793d6-service-ca\") pod \"console-9b4885c74-tq47h\" (UID: \"7fb4d384-4658-48c6-a920-4d5b878793d6\") " pod="openshift-console/console-9b4885c74-tq47h" Apr 16 14:01:58.495291 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:58.495166 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fb4d384-4658-48c6-a920-4d5b878793d6-trusted-ca-bundle\") pod \"console-9b4885c74-tq47h\" (UID: \"7fb4d384-4658-48c6-a920-4d5b878793d6\") " pod="openshift-console/console-9b4885c74-tq47h" Apr 16 14:01:58.495291 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:58.495213 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7fb4d384-4658-48c6-a920-4d5b878793d6-console-oauth-config\") pod \"console-9b4885c74-tq47h\" (UID: \"7fb4d384-4658-48c6-a920-4d5b878793d6\") " pod="openshift-console/console-9b4885c74-tq47h" Apr 16 14:01:58.495291 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:58.495235 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7fb4d384-4658-48c6-a920-4d5b878793d6-console-config\") pod \"console-9b4885c74-tq47h\" (UID: \"7fb4d384-4658-48c6-a920-4d5b878793d6\") " pod="openshift-console/console-9b4885c74-tq47h" Apr 16 14:01:58.495291 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:58.495262 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7fb4d384-4658-48c6-a920-4d5b878793d6-console-serving-cert\") pod \"console-9b4885c74-tq47h\" (UID: \"7fb4d384-4658-48c6-a920-4d5b878793d6\") " pod="openshift-console/console-9b4885c74-tq47h" Apr 16 14:01:58.595879 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:58.595847 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fb4d384-4658-48c6-a920-4d5b878793d6-trusted-ca-bundle\") pod \"console-9b4885c74-tq47h\" (UID: \"7fb4d384-4658-48c6-a920-4d5b878793d6\") " pod="openshift-console/console-9b4885c74-tq47h" Apr 16 14:01:58.596049 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:58.595895 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7fb4d384-4658-48c6-a920-4d5b878793d6-console-oauth-config\") pod \"console-9b4885c74-tq47h\" (UID: \"7fb4d384-4658-48c6-a920-4d5b878793d6\") " pod="openshift-console/console-9b4885c74-tq47h" Apr 16 14:01:58.596049 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:58.595912 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7fb4d384-4658-48c6-a920-4d5b878793d6-console-config\") pod \"console-9b4885c74-tq47h\" (UID: \"7fb4d384-4658-48c6-a920-4d5b878793d6\") " pod="openshift-console/console-9b4885c74-tq47h" Apr 16 14:01:58.596049 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:58.595930 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7fb4d384-4658-48c6-a920-4d5b878793d6-console-serving-cert\") pod \"console-9b4885c74-tq47h\" (UID: \"7fb4d384-4658-48c6-a920-4d5b878793d6\") " pod="openshift-console/console-9b4885c74-tq47h" Apr 16 14:01:58.596049 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:58.595973 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7fb4d384-4658-48c6-a920-4d5b878793d6-oauth-serving-cert\") pod \"console-9b4885c74-tq47h\" (UID: \"7fb4d384-4658-48c6-a920-4d5b878793d6\") " pod="openshift-console/console-9b4885c74-tq47h" Apr 16 14:01:58.596049 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:58.595994 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4x2dc\" (UniqueName: \"kubernetes.io/projected/7fb4d384-4658-48c6-a920-4d5b878793d6-kube-api-access-4x2dc\") pod \"console-9b4885c74-tq47h\" (UID: \"7fb4d384-4658-48c6-a920-4d5b878793d6\") " pod="openshift-console/console-9b4885c74-tq47h" Apr 16 14:01:58.596049 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:58.596014 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7fb4d384-4658-48c6-a920-4d5b878793d6-service-ca\") pod \"console-9b4885c74-tq47h\" (UID: \"7fb4d384-4658-48c6-a920-4d5b878793d6\") " pod="openshift-console/console-9b4885c74-tq47h" Apr 16 14:01:58.596872 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:58.596840 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7fb4d384-4658-48c6-a920-4d5b878793d6-oauth-serving-cert\") pod \"console-9b4885c74-tq47h\" (UID: \"7fb4d384-4658-48c6-a920-4d5b878793d6\") " pod="openshift-console/console-9b4885c74-tq47h" Apr 16 14:01:58.597009 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:58.596880 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7fb4d384-4658-48c6-a920-4d5b878793d6-service-ca\") pod \"console-9b4885c74-tq47h\" (UID: \"7fb4d384-4658-48c6-a920-4d5b878793d6\") " pod="openshift-console/console-9b4885c74-tq47h" Apr 16 14:01:58.597009 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:58.596953 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7fb4d384-4658-48c6-a920-4d5b878793d6-console-config\") pod \"console-9b4885c74-tq47h\" (UID: \"7fb4d384-4658-48c6-a920-4d5b878793d6\") " pod="openshift-console/console-9b4885c74-tq47h" Apr 16 14:01:58.597009 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:58.596973 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fb4d384-4658-48c6-a920-4d5b878793d6-trusted-ca-bundle\") pod \"console-9b4885c74-tq47h\" (UID: \"7fb4d384-4658-48c6-a920-4d5b878793d6\") " pod="openshift-console/console-9b4885c74-tq47h" Apr 16 14:01:58.598327 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:58.598306 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7fb4d384-4658-48c6-a920-4d5b878793d6-console-oauth-config\") pod \"console-9b4885c74-tq47h\" (UID: \"7fb4d384-4658-48c6-a920-4d5b878793d6\") " pod="openshift-console/console-9b4885c74-tq47h" Apr 16 14:01:58.598448 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:58.598430 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7fb4d384-4658-48c6-a920-4d5b878793d6-console-serving-cert\") pod \"console-9b4885c74-tq47h\" (UID: \"7fb4d384-4658-48c6-a920-4d5b878793d6\") " pod="openshift-console/console-9b4885c74-tq47h" Apr 16 14:01:58.604217 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:58.604183 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x2dc\" (UniqueName: \"kubernetes.io/projected/7fb4d384-4658-48c6-a920-4d5b878793d6-kube-api-access-4x2dc\") pod \"console-9b4885c74-tq47h\" (UID: \"7fb4d384-4658-48c6-a920-4d5b878793d6\") " pod="openshift-console/console-9b4885c74-tq47h" Apr 16 14:01:58.623116 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:58.623084 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xjpgz" event={"ID":"7ee1c9f7-5758-4734-a7b3-ea621cc6df95","Type":"ContainerStarted","Data":"538fd68433485ace141e9f965e5863d82f97397f5c22206713d3da0765a15933"} Apr 16 14:01:58.624587 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:58.624568 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lnfdm" event={"ID":"eded8e93-5cbc-435c-aab3-a909812ba63b","Type":"ContainerStarted","Data":"84916060c1dbdb2c1cba5e321f544079f7d08866c98784f760a0f6187fa35f11"} Apr 16 14:01:58.624657 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:58.624595 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lnfdm" event={"ID":"eded8e93-5cbc-435c-aab3-a909812ba63b","Type":"ContainerStarted","Data":"bfdfff86e135f8e78cb8078188dc96fc0d4fe3d73f87e00f7169730732c9478b"} Apr 16 14:01:58.624799 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:58.624789 2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-lnfdm" Apr 16 14:01:58.642704 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:58.642665 2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xjpgz" podStartSLOduration=251.200482112 podStartE2EDuration="4m13.642654393s" podCreationTimestamp="2026-04-16 13:57:45 +0000 UTC" firstStartedPulling="2026-04-16 14:01:55.861962904 +0000 UTC m=+285.691697751" lastFinishedPulling="2026-04-16 14:01:58.30413518 +0000 UTC m=+288.133870032" observedRunningTime="2026-04-16 14:01:58.641819915 +0000 UTC m=+288.471554785" watchObservedRunningTime="2026-04-16 14:01:58.642654393 +0000 UTC m=+288.472389262" Apr 16 14:01:58.655950 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:58.655931 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9b4885c74-tq47h" Apr 16 14:01:58.659333 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:58.659288 2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-lnfdm" podStartSLOduration=251.095269352 podStartE2EDuration="4m13.659275021s" podCreationTimestamp="2026-04-16 13:57:45 +0000 UTC" firstStartedPulling="2026-04-16 14:01:55.737639885 +0000 UTC m=+285.567374734" lastFinishedPulling="2026-04-16 14:01:58.301645556 +0000 UTC m=+288.131380403" observedRunningTime="2026-04-16 14:01:58.657284128 +0000 UTC m=+288.487019019" watchObservedRunningTime="2026-04-16 14:01:58.659275021 +0000 UTC m=+288.489009893" Apr 16 14:01:58.781552 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:58.781394 2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9b4885c74-tq47h"] Apr 16 14:01:58.783954 ip-10-0-136-114 kubenswrapper[2574]: W0416 14:01:58.783930 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fb4d384_4658_48c6_a920_4d5b878793d6.slice/crio-e2bea726be868829ca6abb842aec09bcb52b47a2786192d14d6906aae890a5ff WatchSource:0}: Error finding container e2bea726be868829ca6abb842aec09bcb52b47a2786192d14d6906aae890a5ff: Status 404 returned error can't find the container with id e2bea726be868829ca6abb842aec09bcb52b47a2786192d14d6906aae890a5ff Apr 16 14:01:59.628840 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:59.628800 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9b4885c74-tq47h" event={"ID":"7fb4d384-4658-48c6-a920-4d5b878793d6","Type":"ContainerStarted","Data":"b0020369287e1f8c45d5bd84016002195d0d07312c60695143e40a2d50578c6c"} Apr 16 14:01:59.628840 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:59.628842 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9b4885c74-tq47h" event={"ID":"7fb4d384-4658-48c6-a920-4d5b878793d6","Type":"ContainerStarted","Data":"e2bea726be868829ca6abb842aec09bcb52b47a2786192d14d6906aae890a5ff"} Apr 16 14:01:59.646686 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:01:59.646637 2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-9b4885c74-tq47h" podStartSLOduration=1.6466197230000001 podStartE2EDuration="1.646619723s" podCreationTimestamp="2026-04-16 14:01:58 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 14:01:59.645052968 +0000 UTC m=+289.474787859" watchObservedRunningTime="2026-04-16 14:01:59.646619723 +0000 UTC m=+289.476354595" Apr 16 14:02:08.631210 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:02:08.631180 2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-lnfdm" Apr 16 14:02:08.656222 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:02:08.656191 2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-9b4885c74-tq47h" Apr 16 14:02:08.656222 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:02:08.656232 2574 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-9b4885c74-tq47h" Apr 16 14:02:08.662480 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:02:08.662454 2574 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-9b4885c74-tq47h" Apr 16 14:02:09.663970 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:02:09.663941 2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-9b4885c74-tq47h" Apr 16 14:02:09.710319 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:02:09.710281 2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d646bb8f8-wzpkq"] Apr 16 14:02:10.674518 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:02:10.674485 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6667474d89-tj5v6_5b9fe9ed-1a79-4590-ad6e-74acb8e02aed/cluster-monitoring-operator/0.log" Apr 16 14:02:10.675015 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:02:10.674724 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6667474d89-tj5v6_5b9fe9ed-1a79-4590-ad6e-74acb8e02aed/cluster-monitoring-operator/0.log" Apr 16 14:02:10.686234 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:02:10.686216 2574 kubelet.go:1628] "Image garbage collection succeeded" Apr 16 14:02:34.731010 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:02:34.730942 2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-6d646bb8f8-wzpkq" podUID="4e213a29-44bc-448b-a4e4-b77a596aeb54" containerName="console" containerID="cri-o://16f64eebb681da7a9420c4320890641e6b816a9ca68c90db5c768e2eaa0d3705" gracePeriod=15 Apr 16 14:02:34.986552 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:02:34.986494 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6d646bb8f8-wzpkq_4e213a29-44bc-448b-a4e4-b77a596aeb54/console/0.log" Apr 16 14:02:34.986676 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:02:34.986557 2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d646bb8f8-wzpkq" Apr 16 14:02:35.111122 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:02:35.111079 2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sjjr4\" (UniqueName: \"kubernetes.io/projected/4e213a29-44bc-448b-a4e4-b77a596aeb54-kube-api-access-sjjr4\") pod \"4e213a29-44bc-448b-a4e4-b77a596aeb54\" (UID: \"4e213a29-44bc-448b-a4e4-b77a596aeb54\") " Apr 16 14:02:35.111322 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:02:35.111136 2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4e213a29-44bc-448b-a4e4-b77a596aeb54-service-ca\") pod \"4e213a29-44bc-448b-a4e4-b77a596aeb54\" (UID: \"4e213a29-44bc-448b-a4e4-b77a596aeb54\") " Apr 16 14:02:35.111322 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:02:35.111155 2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e213a29-44bc-448b-a4e4-b77a596aeb54-trusted-ca-bundle\") pod \"4e213a29-44bc-448b-a4e4-b77a596aeb54\" (UID: \"4e213a29-44bc-448b-a4e4-b77a596aeb54\") " Apr 16 14:02:35.111322 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:02:35.111178 2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4e213a29-44bc-448b-a4e4-b77a596aeb54-console-oauth-config\") pod \"4e213a29-44bc-448b-a4e4-b77a596aeb54\" (UID: \"4e213a29-44bc-448b-a4e4-b77a596aeb54\") " Apr 16 14:02:35.111322 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:02:35.111213 2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4e213a29-44bc-448b-a4e4-b77a596aeb54-oauth-serving-cert\") pod \"4e213a29-44bc-448b-a4e4-b77a596aeb54\" (UID: \"4e213a29-44bc-448b-a4e4-b77a596aeb54\") " Apr 16 14:02:35.111537 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:02:35.111329 2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4e213a29-44bc-448b-a4e4-b77a596aeb54-console-config\") pod \"4e213a29-44bc-448b-a4e4-b77a596aeb54\" (UID: \"4e213a29-44bc-448b-a4e4-b77a596aeb54\") " Apr 16 14:02:35.111537 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:02:35.111386 2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4e213a29-44bc-448b-a4e4-b77a596aeb54-console-serving-cert\") pod \"4e213a29-44bc-448b-a4e4-b77a596aeb54\" (UID: \"4e213a29-44bc-448b-a4e4-b77a596aeb54\") " Apr 16 14:02:35.111702 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:02:35.111666 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e213a29-44bc-448b-a4e4-b77a596aeb54-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4e213a29-44bc-448b-a4e4-b77a596aeb54" (UID: "4e213a29-44bc-448b-a4e4-b77a596aeb54"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 16 14:02:35.111702 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:02:35.111677 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e213a29-44bc-448b-a4e4-b77a596aeb54-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "4e213a29-44bc-448b-a4e4-b77a596aeb54" (UID: "4e213a29-44bc-448b-a4e4-b77a596aeb54"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 16 14:02:35.111851 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:02:35.111672 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e213a29-44bc-448b-a4e4-b77a596aeb54-service-ca" (OuterVolumeSpecName: "service-ca") pod "4e213a29-44bc-448b-a4e4-b77a596aeb54" (UID: "4e213a29-44bc-448b-a4e4-b77a596aeb54"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 16 14:02:35.111851 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:02:35.111701 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e213a29-44bc-448b-a4e4-b77a596aeb54-console-config" (OuterVolumeSpecName: "console-config") pod "4e213a29-44bc-448b-a4e4-b77a596aeb54" (UID: "4e213a29-44bc-448b-a4e4-b77a596aeb54"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 16 14:02:35.113340 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:02:35.113318 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e213a29-44bc-448b-a4e4-b77a596aeb54-kube-api-access-sjjr4" (OuterVolumeSpecName: "kube-api-access-sjjr4") pod "4e213a29-44bc-448b-a4e4-b77a596aeb54" (UID: "4e213a29-44bc-448b-a4e4-b77a596aeb54"). InnerVolumeSpecName "kube-api-access-sjjr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 16 14:02:35.113461 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:02:35.113424 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e213a29-44bc-448b-a4e4-b77a596aeb54-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4e213a29-44bc-448b-a4e4-b77a596aeb54" (UID: "4e213a29-44bc-448b-a4e4-b77a596aeb54"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 16 14:02:35.113510 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:02:35.113448 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e213a29-44bc-448b-a4e4-b77a596aeb54-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4e213a29-44bc-448b-a4e4-b77a596aeb54" (UID: "4e213a29-44bc-448b-a4e4-b77a596aeb54"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 16 14:02:35.212255 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:02:35.212217 2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4e213a29-44bc-448b-a4e4-b77a596aeb54-console-oauth-config\") on node \"ip-10-0-136-114.ec2.internal\" DevicePath \"\"" Apr 16 14:02:35.212255 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:02:35.212252 2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4e213a29-44bc-448b-a4e4-b77a596aeb54-oauth-serving-cert\") on node \"ip-10-0-136-114.ec2.internal\" DevicePath \"\"" Apr 16 14:02:35.212255 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:02:35.212261 2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4e213a29-44bc-448b-a4e4-b77a596aeb54-console-config\") on node \"ip-10-0-136-114.ec2.internal\" DevicePath \"\"" Apr 16 14:02:35.212487 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:02:35.212270 2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4e213a29-44bc-448b-a4e4-b77a596aeb54-console-serving-cert\") on node \"ip-10-0-136-114.ec2.internal\" DevicePath \"\"" Apr 16 14:02:35.212487 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:02:35.212283 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjjr4\" (UniqueName: \"kubernetes.io/projected/4e213a29-44bc-448b-a4e4-b77a596aeb54-kube-api-access-sjjr4\") on node \"ip-10-0-136-114.ec2.internal\" DevicePath \"\"" Apr 16 14:02:35.212487 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:02:35.212297 2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4e213a29-44bc-448b-a4e4-b77a596aeb54-service-ca\") on node \"ip-10-0-136-114.ec2.internal\" DevicePath \"\"" Apr 16 14:02:35.212487 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:02:35.212308 2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e213a29-44bc-448b-a4e4-b77a596aeb54-trusted-ca-bundle\") on node \"ip-10-0-136-114.ec2.internal\" DevicePath \"\"" Apr 16 14:02:35.736986 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:02:35.736959 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6d646bb8f8-wzpkq_4e213a29-44bc-448b-a4e4-b77a596aeb54/console/0.log" Apr 16 14:02:35.737447 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:02:35.736998 2574 generic.go:358] "Generic (PLEG): container finished" podID="4e213a29-44bc-448b-a4e4-b77a596aeb54" containerID="16f64eebb681da7a9420c4320890641e6b816a9ca68c90db5c768e2eaa0d3705" exitCode=2 Apr 16 14:02:35.737447 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:02:35.737031 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d646bb8f8-wzpkq" event={"ID":"4e213a29-44bc-448b-a4e4-b77a596aeb54","Type":"ContainerDied","Data":"16f64eebb681da7a9420c4320890641e6b816a9ca68c90db5c768e2eaa0d3705"} Apr 16 14:02:35.737447 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:02:35.737071 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d646bb8f8-wzpkq" event={"ID":"4e213a29-44bc-448b-a4e4-b77a596aeb54","Type":"ContainerDied","Data":"64b48d1dfb57e77b457d54cbda61963ddc4efe363119b9d23a401996f91c3e55"} Apr 16 14:02:35.737447 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:02:35.737076 2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d646bb8f8-wzpkq" Apr 16 14:02:35.737447 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:02:35.737087 2574 scope.go:117] "RemoveContainer" containerID="16f64eebb681da7a9420c4320890641e6b816a9ca68c90db5c768e2eaa0d3705" Apr 16 14:02:35.745550 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:02:35.745535 2574 scope.go:117] "RemoveContainer" containerID="16f64eebb681da7a9420c4320890641e6b816a9ca68c90db5c768e2eaa0d3705" Apr 16 14:02:35.745811 ip-10-0-136-114 kubenswrapper[2574]: E0416 14:02:35.745790 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16f64eebb681da7a9420c4320890641e6b816a9ca68c90db5c768e2eaa0d3705\": container with ID starting with 16f64eebb681da7a9420c4320890641e6b816a9ca68c90db5c768e2eaa0d3705 not found: ID does not exist" containerID="16f64eebb681da7a9420c4320890641e6b816a9ca68c90db5c768e2eaa0d3705" Apr 16 14:02:35.745864 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:02:35.745819 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16f64eebb681da7a9420c4320890641e6b816a9ca68c90db5c768e2eaa0d3705"} err="failed to get container status \"16f64eebb681da7a9420c4320890641e6b816a9ca68c90db5c768e2eaa0d3705\": rpc error: code = NotFound desc = could not find container \"16f64eebb681da7a9420c4320890641e6b816a9ca68c90db5c768e2eaa0d3705\": container with ID starting with 16f64eebb681da7a9420c4320890641e6b816a9ca68c90db5c768e2eaa0d3705 not found: ID does not exist" Apr 16 14:02:35.757838 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:02:35.757817 2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d646bb8f8-wzpkq"] Apr 16 14:02:35.761582 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:02:35.761560 2574 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6d646bb8f8-wzpkq"] Apr 16 14:02:36.809381 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:02:36.809342 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e213a29-44bc-448b-a4e4-b77a596aeb54" path="/var/lib/kubelet/pods/4e213a29-44bc-448b-a4e4-b77a596aeb54/volumes" Apr 16 14:03:14.191579 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:14.191549 2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-695dd5755c-lkc7t"] Apr 16 14:03:14.191994 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:14.191886 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e213a29-44bc-448b-a4e4-b77a596aeb54" containerName="console" Apr 16 14:03:14.191994 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:14.191900 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e213a29-44bc-448b-a4e4-b77a596aeb54" containerName="console" Apr 16 14:03:14.191994 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:14.191954 2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="4e213a29-44bc-448b-a4e4-b77a596aeb54" containerName="console" Apr 16 14:03:14.194566 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:14.194547 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-695dd5755c-lkc7t" Apr 16 14:03:14.207291 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:14.207268 2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-695dd5755c-lkc7t"] Apr 16 14:03:14.243689 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:14.243659 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65m7x\" (UniqueName: \"kubernetes.io/projected/68515d67-48e0-4d3f-bc23-bfe7fbb9cc63-kube-api-access-65m7x\") pod \"console-695dd5755c-lkc7t\" (UID: \"68515d67-48e0-4d3f-bc23-bfe7fbb9cc63\") " pod="openshift-console/console-695dd5755c-lkc7t" Apr 16 14:03:14.243836 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:14.243693 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/68515d67-48e0-4d3f-bc23-bfe7fbb9cc63-oauth-serving-cert\") pod \"console-695dd5755c-lkc7t\" (UID: \"68515d67-48e0-4d3f-bc23-bfe7fbb9cc63\") " pod="openshift-console/console-695dd5755c-lkc7t" Apr 16 14:03:14.243836 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:14.243710 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/68515d67-48e0-4d3f-bc23-bfe7fbb9cc63-console-config\") pod \"console-695dd5755c-lkc7t\" (UID: \"68515d67-48e0-4d3f-bc23-bfe7fbb9cc63\") " pod="openshift-console/console-695dd5755c-lkc7t" Apr 16 14:03:14.243836 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:14.243732 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68515d67-48e0-4d3f-bc23-bfe7fbb9cc63-trusted-ca-bundle\") pod \"console-695dd5755c-lkc7t\" (UID: \"68515d67-48e0-4d3f-bc23-bfe7fbb9cc63\") " pod="openshift-console/console-695dd5755c-lkc7t" Apr 16 14:03:14.244003 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:14.243843 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/68515d67-48e0-4d3f-bc23-bfe7fbb9cc63-console-oauth-config\") pod \"console-695dd5755c-lkc7t\" (UID: \"68515d67-48e0-4d3f-bc23-bfe7fbb9cc63\") " pod="openshift-console/console-695dd5755c-lkc7t" Apr 16 14:03:14.244003 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:14.243879 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/68515d67-48e0-4d3f-bc23-bfe7fbb9cc63-console-serving-cert\") pod \"console-695dd5755c-lkc7t\" (UID: \"68515d67-48e0-4d3f-bc23-bfe7fbb9cc63\") " pod="openshift-console/console-695dd5755c-lkc7t" Apr 16 14:03:14.244003 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:14.243945 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/68515d67-48e0-4d3f-bc23-bfe7fbb9cc63-service-ca\") pod \"console-695dd5755c-lkc7t\" (UID: \"68515d67-48e0-4d3f-bc23-bfe7fbb9cc63\") " pod="openshift-console/console-695dd5755c-lkc7t" Apr 16 14:03:14.344349 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:14.344318 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-65m7x\" (UniqueName: \"kubernetes.io/projected/68515d67-48e0-4d3f-bc23-bfe7fbb9cc63-kube-api-access-65m7x\") pod \"console-695dd5755c-lkc7t\" (UID: \"68515d67-48e0-4d3f-bc23-bfe7fbb9cc63\") " pod="openshift-console/console-695dd5755c-lkc7t" Apr 16 14:03:14.344349 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:14.344350 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/68515d67-48e0-4d3f-bc23-bfe7fbb9cc63-oauth-serving-cert\") pod \"console-695dd5755c-lkc7t\" (UID: \"68515d67-48e0-4d3f-bc23-bfe7fbb9cc63\") " pod="openshift-console/console-695dd5755c-lkc7t" Apr 16 14:03:14.344565 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:14.344367 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/68515d67-48e0-4d3f-bc23-bfe7fbb9cc63-console-config\") pod \"console-695dd5755c-lkc7t\" (UID: \"68515d67-48e0-4d3f-bc23-bfe7fbb9cc63\") " pod="openshift-console/console-695dd5755c-lkc7t" Apr 16 14:03:14.344565 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:14.344389 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68515d67-48e0-4d3f-bc23-bfe7fbb9cc63-trusted-ca-bundle\") pod \"console-695dd5755c-lkc7t\" (UID: \"68515d67-48e0-4d3f-bc23-bfe7fbb9cc63\") " pod="openshift-console/console-695dd5755c-lkc7t" Apr 16 14:03:14.344565 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:14.344418 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/68515d67-48e0-4d3f-bc23-bfe7fbb9cc63-console-oauth-config\") pod \"console-695dd5755c-lkc7t\" (UID: \"68515d67-48e0-4d3f-bc23-bfe7fbb9cc63\") " pod="openshift-console/console-695dd5755c-lkc7t" Apr 16 14:03:14.344565 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:14.344435 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/68515d67-48e0-4d3f-bc23-bfe7fbb9cc63-console-serving-cert\") pod \"console-695dd5755c-lkc7t\" (UID: \"68515d67-48e0-4d3f-bc23-bfe7fbb9cc63\") " pod="openshift-console/console-695dd5755c-lkc7t" Apr 16 14:03:14.345011 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:14.344975 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/68515d67-48e0-4d3f-bc23-bfe7fbb9cc63-service-ca\") pod \"console-695dd5755c-lkc7t\" (UID: \"68515d67-48e0-4d3f-bc23-bfe7fbb9cc63\") " pod="openshift-console/console-695dd5755c-lkc7t" Apr 16 14:03:14.345563 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:14.345535 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/68515d67-48e0-4d3f-bc23-bfe7fbb9cc63-oauth-serving-cert\") pod \"console-695dd5755c-lkc7t\" (UID: \"68515d67-48e0-4d3f-bc23-bfe7fbb9cc63\") " pod="openshift-console/console-695dd5755c-lkc7t" Apr 16 14:03:14.345718 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:14.345691 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/68515d67-48e0-4d3f-bc23-bfe7fbb9cc63-console-config\") pod \"console-695dd5755c-lkc7t\" (UID: \"68515d67-48e0-4d3f-bc23-bfe7fbb9cc63\") " pod="openshift-console/console-695dd5755c-lkc7t" Apr 16 14:03:14.345850 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:14.345827 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/68515d67-48e0-4d3f-bc23-bfe7fbb9cc63-service-ca\") pod \"console-695dd5755c-lkc7t\" (UID: \"68515d67-48e0-4d3f-bc23-bfe7fbb9cc63\") " pod="openshift-console/console-695dd5755c-lkc7t" Apr 16 14:03:14.351206 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:14.347494 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68515d67-48e0-4d3f-bc23-bfe7fbb9cc63-trusted-ca-bundle\") pod \"console-695dd5755c-lkc7t\" (UID: \"68515d67-48e0-4d3f-bc23-bfe7fbb9cc63\") " pod="openshift-console/console-695dd5755c-lkc7t" Apr 16 14:03:14.351206 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:14.347942 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/68515d67-48e0-4d3f-bc23-bfe7fbb9cc63-console-oauth-config\") pod \"console-695dd5755c-lkc7t\" (UID: \"68515d67-48e0-4d3f-bc23-bfe7fbb9cc63\") " pod="openshift-console/console-695dd5755c-lkc7t" Apr 16 14:03:14.352205 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:14.352181 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/68515d67-48e0-4d3f-bc23-bfe7fbb9cc63-console-serving-cert\") pod \"console-695dd5755c-lkc7t\" (UID: \"68515d67-48e0-4d3f-bc23-bfe7fbb9cc63\") " pod="openshift-console/console-695dd5755c-lkc7t" Apr 16 14:03:14.353621 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:14.353603 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65m7x\" (UniqueName: \"kubernetes.io/projected/68515d67-48e0-4d3f-bc23-bfe7fbb9cc63-kube-api-access-65m7x\") pod \"console-695dd5755c-lkc7t\" (UID: \"68515d67-48e0-4d3f-bc23-bfe7fbb9cc63\") " pod="openshift-console/console-695dd5755c-lkc7t" Apr 16 14:03:14.504521 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:14.504420 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-695dd5755c-lkc7t" Apr 16 14:03:14.622693 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:14.622654 2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-695dd5755c-lkc7t"] Apr 16 14:03:14.625522 ip-10-0-136-114 kubenswrapper[2574]: W0416 14:03:14.625492 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68515d67_48e0_4d3f_bc23_bfe7fbb9cc63.slice/crio-259700f9276db4d8a7c9431571d25de96ef072f526d439d00cc45198cfa7544b WatchSource:0}: Error finding container 259700f9276db4d8a7c9431571d25de96ef072f526d439d00cc45198cfa7544b: Status 404 returned error can't find the container with id 259700f9276db4d8a7c9431571d25de96ef072f526d439d00cc45198cfa7544b Apr 16 14:03:14.627458 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:14.627438 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Apr 16 14:03:14.850849 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:14.850814 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-695dd5755c-lkc7t" event={"ID":"68515d67-48e0-4d3f-bc23-bfe7fbb9cc63","Type":"ContainerStarted","Data":"c2cc5ab609627f9e810a8ff49a6b486adb0aaf7ab9fcf9351bcbd6e27855395d"} Apr 16 14:03:14.850849 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:14.850848 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-695dd5755c-lkc7t" event={"ID":"68515d67-48e0-4d3f-bc23-bfe7fbb9cc63","Type":"ContainerStarted","Data":"259700f9276db4d8a7c9431571d25de96ef072f526d439d00cc45198cfa7544b"} Apr 16 14:03:14.869306 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:14.869254 2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-695dd5755c-lkc7t" podStartSLOduration=0.869237065 podStartE2EDuration="869.237065ms" podCreationTimestamp="2026-04-16 14:03:14 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 14:03:14.868049576 +0000 UTC m=+364.697784446" watchObservedRunningTime="2026-04-16 14:03:14.869237065 +0000 UTC m=+364.698971936" Apr 16 14:03:24.505320 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:24.505282 2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-695dd5755c-lkc7t" Apr 16 14:03:24.505320 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:24.505328 2574 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-695dd5755c-lkc7t" Apr 16 14:03:24.510191 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:24.510169 2574 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-695dd5755c-lkc7t" Apr 16 14:03:24.881504 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:24.881477 2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-695dd5755c-lkc7t" Apr 16 14:03:24.931522 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:24.931483 2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9b4885c74-tq47h"] Apr 16 14:03:49.951829 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:49.951773 2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-9b4885c74-tq47h" podUID="7fb4d384-4658-48c6-a920-4d5b878793d6" containerName="console" containerID="cri-o://b0020369287e1f8c45d5bd84016002195d0d07312c60695143e40a2d50578c6c" gracePeriod=15 Apr 16 14:03:50.187582 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:50.187558 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-9b4885c74-tq47h_7fb4d384-4658-48c6-a920-4d5b878793d6/console/0.log" Apr 16 14:03:50.187690 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:50.187620 2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9b4885c74-tq47h" Apr 16 14:03:50.231341 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:50.231274 2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7fb4d384-4658-48c6-a920-4d5b878793d6-console-serving-cert\") pod \"7fb4d384-4658-48c6-a920-4d5b878793d6\" (UID: \"7fb4d384-4658-48c6-a920-4d5b878793d6\") " Apr 16 14:03:50.231341 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:50.231326 2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4x2dc\" (UniqueName: \"kubernetes.io/projected/7fb4d384-4658-48c6-a920-4d5b878793d6-kube-api-access-4x2dc\") pod \"7fb4d384-4658-48c6-a920-4d5b878793d6\" (UID: \"7fb4d384-4658-48c6-a920-4d5b878793d6\") " Apr 16 14:03:50.231517 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:50.231356 2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7fb4d384-4658-48c6-a920-4d5b878793d6-console-oauth-config\") pod \"7fb4d384-4658-48c6-a920-4d5b878793d6\" (UID: \"7fb4d384-4658-48c6-a920-4d5b878793d6\") " Apr 16 14:03:50.231517 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:50.231390 2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7fb4d384-4658-48c6-a920-4d5b878793d6-oauth-serving-cert\") pod \"7fb4d384-4658-48c6-a920-4d5b878793d6\" (UID: \"7fb4d384-4658-48c6-a920-4d5b878793d6\") " Apr 16 14:03:50.231517 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:50.231427 2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7fb4d384-4658-48c6-a920-4d5b878793d6-service-ca\") pod \"7fb4d384-4658-48c6-a920-4d5b878793d6\" (UID: \"7fb4d384-4658-48c6-a920-4d5b878793d6\") " Apr 16 14:03:50.231517 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:50.231454 2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7fb4d384-4658-48c6-a920-4d5b878793d6-console-config\") pod \"7fb4d384-4658-48c6-a920-4d5b878793d6\" (UID: \"7fb4d384-4658-48c6-a920-4d5b878793d6\") " Apr 16 14:03:50.231517 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:50.231483 2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fb4d384-4658-48c6-a920-4d5b878793d6-trusted-ca-bundle\") pod \"7fb4d384-4658-48c6-a920-4d5b878793d6\" (UID: \"7fb4d384-4658-48c6-a920-4d5b878793d6\") " Apr 16 14:03:50.231970 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:50.231916 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fb4d384-4658-48c6-a920-4d5b878793d6-console-config" (OuterVolumeSpecName: "console-config") pod "7fb4d384-4658-48c6-a920-4d5b878793d6" (UID: "7fb4d384-4658-48c6-a920-4d5b878793d6"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 16 14:03:50.231970 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:50.231955 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fb4d384-4658-48c6-a920-4d5b878793d6-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7fb4d384-4658-48c6-a920-4d5b878793d6" (UID: "7fb4d384-4658-48c6-a920-4d5b878793d6"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 16 14:03:50.232123 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:50.231965 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fb4d384-4658-48c6-a920-4d5b878793d6-service-ca" (OuterVolumeSpecName: "service-ca") pod "7fb4d384-4658-48c6-a920-4d5b878793d6" (UID: "7fb4d384-4658-48c6-a920-4d5b878793d6"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 16 14:03:50.232123 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:50.232042 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fb4d384-4658-48c6-a920-4d5b878793d6-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7fb4d384-4658-48c6-a920-4d5b878793d6" (UID: "7fb4d384-4658-48c6-a920-4d5b878793d6"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 16 14:03:50.233437 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:50.233416 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fb4d384-4658-48c6-a920-4d5b878793d6-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7fb4d384-4658-48c6-a920-4d5b878793d6" (UID: "7fb4d384-4658-48c6-a920-4d5b878793d6"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 16 14:03:50.233518 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:50.233438 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fb4d384-4658-48c6-a920-4d5b878793d6-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7fb4d384-4658-48c6-a920-4d5b878793d6" (UID: "7fb4d384-4658-48c6-a920-4d5b878793d6"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 16 14:03:50.233567 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:50.233519 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fb4d384-4658-48c6-a920-4d5b878793d6-kube-api-access-4x2dc" (OuterVolumeSpecName: "kube-api-access-4x2dc") pod "7fb4d384-4658-48c6-a920-4d5b878793d6" (UID: "7fb4d384-4658-48c6-a920-4d5b878793d6"). InnerVolumeSpecName "kube-api-access-4x2dc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 16 14:03:50.332158 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:50.332124 2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7fb4d384-4658-48c6-a920-4d5b878793d6-oauth-serving-cert\") on node \"ip-10-0-136-114.ec2.internal\" DevicePath \"\"" Apr 16 14:03:50.332158 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:50.332149 2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7fb4d384-4658-48c6-a920-4d5b878793d6-service-ca\") on node \"ip-10-0-136-114.ec2.internal\" DevicePath \"\"" Apr 16 14:03:50.332158 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:50.332159 2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7fb4d384-4658-48c6-a920-4d5b878793d6-console-config\") on node \"ip-10-0-136-114.ec2.internal\" DevicePath \"\"" Apr 16 14:03:50.332399 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:50.332167 2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fb4d384-4658-48c6-a920-4d5b878793d6-trusted-ca-bundle\") on node \"ip-10-0-136-114.ec2.internal\" DevicePath \"\"" Apr 16 14:03:50.332399 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:50.332177 2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7fb4d384-4658-48c6-a920-4d5b878793d6-console-serving-cert\") on node \"ip-10-0-136-114.ec2.internal\" DevicePath \"\"" Apr 16 14:03:50.332399 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:50.332186 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4x2dc\" (UniqueName: \"kubernetes.io/projected/7fb4d384-4658-48c6-a920-4d5b878793d6-kube-api-access-4x2dc\") on node \"ip-10-0-136-114.ec2.internal\" DevicePath \"\"" Apr 16 14:03:50.332399 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:50.332196 2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7fb4d384-4658-48c6-a920-4d5b878793d6-console-oauth-config\") on node \"ip-10-0-136-114.ec2.internal\" DevicePath \"\"" Apr 16 14:03:50.951470 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:50.951442 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-9b4885c74-tq47h_7fb4d384-4658-48c6-a920-4d5b878793d6/console/0.log" Apr 16 14:03:50.951616 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:50.951480 2574 generic.go:358] "Generic (PLEG): container finished" podID="7fb4d384-4658-48c6-a920-4d5b878793d6" containerID="b0020369287e1f8c45d5bd84016002195d0d07312c60695143e40a2d50578c6c" exitCode=2 Apr 16 14:03:50.951616 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:50.951522 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9b4885c74-tq47h" event={"ID":"7fb4d384-4658-48c6-a920-4d5b878793d6","Type":"ContainerDied","Data":"b0020369287e1f8c45d5bd84016002195d0d07312c60695143e40a2d50578c6c"} Apr 16 14:03:50.951616 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:50.951544 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9b4885c74-tq47h" event={"ID":"7fb4d384-4658-48c6-a920-4d5b878793d6","Type":"ContainerDied","Data":"e2bea726be868829ca6abb842aec09bcb52b47a2786192d14d6906aae890a5ff"} Apr 16 14:03:50.951616 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:50.951542 2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9b4885c74-tq47h" Apr 16 14:03:50.951616 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:50.951555 2574 scope.go:117] "RemoveContainer" containerID="b0020369287e1f8c45d5bd84016002195d0d07312c60695143e40a2d50578c6c" Apr 16 14:03:50.959246 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:50.959227 2574 scope.go:117] "RemoveContainer" containerID="b0020369287e1f8c45d5bd84016002195d0d07312c60695143e40a2d50578c6c" Apr 16 14:03:50.959505 ip-10-0-136-114 kubenswrapper[2574]: E0416 14:03:50.959474 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0020369287e1f8c45d5bd84016002195d0d07312c60695143e40a2d50578c6c\": container with ID starting with b0020369287e1f8c45d5bd84016002195d0d07312c60695143e40a2d50578c6c not found: ID does not exist" containerID="b0020369287e1f8c45d5bd84016002195d0d07312c60695143e40a2d50578c6c" Apr 16 14:03:50.959549 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:50.959494 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0020369287e1f8c45d5bd84016002195d0d07312c60695143e40a2d50578c6c"} err="failed to get container status \"b0020369287e1f8c45d5bd84016002195d0d07312c60695143e40a2d50578c6c\": rpc error: code = NotFound desc = could not find container \"b0020369287e1f8c45d5bd84016002195d0d07312c60695143e40a2d50578c6c\": container with ID starting with b0020369287e1f8c45d5bd84016002195d0d07312c60695143e40a2d50578c6c not found: ID does not exist" Apr 16 14:03:50.968972 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:50.968948 2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9b4885c74-tq47h"] Apr 16 14:03:50.972717 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:50.972692 2574 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-9b4885c74-tq47h"] Apr 16 14:03:52.808446 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:03:52.808409 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fb4d384-4658-48c6-a920-4d5b878793d6" path="/var/lib/kubelet/pods/7fb4d384-4658-48c6-a920-4d5b878793d6/volumes" Apr 16 14:05:15.675020 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:15.674983 2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-5q7r9"] Apr 16 14:05:15.675721 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:15.675422 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fb4d384-4658-48c6-a920-4d5b878793d6" containerName="console" Apr 16 14:05:15.675721 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:15.675439 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fb4d384-4658-48c6-a920-4d5b878793d6" containerName="console" Apr 16 14:05:15.675721 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:15.675516 2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="7fb4d384-4658-48c6-a920-4d5b878793d6" containerName="console" Apr 16 14:05:15.677495 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:15.677479 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-5q7r9" Apr 16 14:05:15.680403 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:15.680383 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-w4gpb\"" Apr 16 14:05:15.680634 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:15.680618 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\"" Apr 16 14:05:15.680702 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:15.680689 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\"" Apr 16 14:05:15.680797 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:15.680773 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\"" Apr 16 14:05:15.699135 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:15.699089 2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-5q7r9"] Apr 16 14:05:15.762175 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:15.762135 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b7v7\" (UniqueName: \"kubernetes.io/projected/180d28e1-662d-4b44-88d5-05d25814bb46-kube-api-access-8b7v7\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-5q7r9\" (UID: \"180d28e1-662d-4b44-88d5-05d25814bb46\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-5q7r9" Apr 16 14:05:15.762349 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:15.762201 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/180d28e1-662d-4b44-88d5-05d25814bb46-certificates\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-5q7r9\" (UID: \"180d28e1-662d-4b44-88d5-05d25814bb46\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-5q7r9" Apr 16 14:05:15.863068 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:15.863034 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8b7v7\" (UniqueName: \"kubernetes.io/projected/180d28e1-662d-4b44-88d5-05d25814bb46-kube-api-access-8b7v7\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-5q7r9\" (UID: \"180d28e1-662d-4b44-88d5-05d25814bb46\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-5q7r9" Apr 16 14:05:15.863248 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:15.863088 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/180d28e1-662d-4b44-88d5-05d25814bb46-certificates\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-5q7r9\" (UID: \"180d28e1-662d-4b44-88d5-05d25814bb46\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-5q7r9" Apr 16 14:05:15.865303 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:15.865285 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/180d28e1-662d-4b44-88d5-05d25814bb46-certificates\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-5q7r9\" (UID: \"180d28e1-662d-4b44-88d5-05d25814bb46\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-5q7r9" Apr 16 14:05:15.872154 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:15.872127 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b7v7\" (UniqueName: \"kubernetes.io/projected/180d28e1-662d-4b44-88d5-05d25814bb46-kube-api-access-8b7v7\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-5q7r9\" (UID: \"180d28e1-662d-4b44-88d5-05d25814bb46\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-5q7r9" Apr 16 14:05:15.988072 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:15.987983 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-5q7r9" Apr 16 14:05:16.110022 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:16.109989 2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-5q7r9"] Apr 16 14:05:16.113792 ip-10-0-136-114 kubenswrapper[2574]: W0416 14:05:16.113762 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod180d28e1_662d_4b44_88d5_05d25814bb46.slice/crio-71513f72f95b350e563b62d9f43d8f10ce69a1181e6774134c474a7898f7270f WatchSource:0}: Error finding container 71513f72f95b350e563b62d9f43d8f10ce69a1181e6774134c474a7898f7270f: Status 404 returned error can't find the container with id 71513f72f95b350e563b62d9f43d8f10ce69a1181e6774134c474a7898f7270f Apr 16 14:05:16.197014 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:16.196979 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-5q7r9" event={"ID":"180d28e1-662d-4b44-88d5-05d25814bb46","Type":"ContainerStarted","Data":"71513f72f95b350e563b62d9f43d8f10ce69a1181e6774134c474a7898f7270f"} Apr 16 14:05:20.728400 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:20.728366 2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-ffbb595cb-njsxc"] Apr 16 14:05:20.730602 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:20.730587 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-ffbb595cb-njsxc" Apr 16 14:05:20.732979 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:20.732948 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\"" Apr 16 14:05:20.733101 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:20.732990 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\"" Apr 16 14:05:20.733196 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:20.733181 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-xlfs9\"" Apr 16 14:05:20.739561 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:20.739538 2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-ffbb595cb-njsxc"] Apr 16 14:05:20.906109 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:20.906080 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/d91d133a-9029-403e-9096-db48df37b7ff-cabundle0\") pod \"keda-operator-ffbb595cb-njsxc\" (UID: \"d91d133a-9029-403e-9096-db48df37b7ff\") " pod="openshift-keda/keda-operator-ffbb595cb-njsxc" Apr 16 14:05:20.906268 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:20.906139 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d91d133a-9029-403e-9096-db48df37b7ff-certificates\") pod \"keda-operator-ffbb595cb-njsxc\" (UID: \"d91d133a-9029-403e-9096-db48df37b7ff\") " pod="openshift-keda/keda-operator-ffbb595cb-njsxc" Apr 16 14:05:20.906268 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:20.906188 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrh57\" (UniqueName: \"kubernetes.io/projected/d91d133a-9029-403e-9096-db48df37b7ff-kube-api-access-jrh57\") pod \"keda-operator-ffbb595cb-njsxc\" (UID: \"d91d133a-9029-403e-9096-db48df37b7ff\") " pod="openshift-keda/keda-operator-ffbb595cb-njsxc" Apr 16 14:05:21.000295 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:21.000224 2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-7c9f485588-k7lfz"] Apr 16 14:05:21.002243 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:21.002225 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-k7lfz" Apr 16 14:05:21.004728 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:21.004707 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\"" Apr 16 14:05:21.006598 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:21.006574 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d91d133a-9029-403e-9096-db48df37b7ff-certificates\") pod \"keda-operator-ffbb595cb-njsxc\" (UID: \"d91d133a-9029-403e-9096-db48df37b7ff\") " pod="openshift-keda/keda-operator-ffbb595cb-njsxc" Apr 16 14:05:21.006669 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:21.006622 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jrh57\" (UniqueName: \"kubernetes.io/projected/d91d133a-9029-403e-9096-db48df37b7ff-kube-api-access-jrh57\") pod \"keda-operator-ffbb595cb-njsxc\" (UID: \"d91d133a-9029-403e-9096-db48df37b7ff\") " pod="openshift-keda/keda-operator-ffbb595cb-njsxc" Apr 16 14:05:21.006725 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:21.006709 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/d91d133a-9029-403e-9096-db48df37b7ff-cabundle0\") pod \"keda-operator-ffbb595cb-njsxc\" (UID: \"d91d133a-9029-403e-9096-db48df37b7ff\") " pod="openshift-keda/keda-operator-ffbb595cb-njsxc" Apr 16 14:05:21.006811 ip-10-0-136-114 kubenswrapper[2574]: E0416 14:05:21.006713 2574 secret.go:281] references non-existent secret key: ca.crt Apr 16 14:05:21.006857 ip-10-0-136-114 kubenswrapper[2574]: E0416 14:05:21.006818 2574 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt Apr 16 14:05:21.006857 ip-10-0-136-114 kubenswrapper[2574]: E0416 14:05:21.006831 2574 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-ffbb595cb-njsxc: references non-existent secret key: ca.crt Apr 16 14:05:21.006945 ip-10-0-136-114 kubenswrapper[2574]: E0416 14:05:21.006879 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d91d133a-9029-403e-9096-db48df37b7ff-certificates podName:d91d133a-9029-403e-9096-db48df37b7ff nodeName:}" failed. No retries permitted until 2026-04-16 14:05:21.506866308 +0000 UTC m=+491.336601156 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/d91d133a-9029-403e-9096-db48df37b7ff-certificates") pod "keda-operator-ffbb595cb-njsxc" (UID: "d91d133a-9029-403e-9096-db48df37b7ff") : references non-existent secret key: ca.crt Apr 16 14:05:21.007253 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:21.007236 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/d91d133a-9029-403e-9096-db48df37b7ff-cabundle0\") pod \"keda-operator-ffbb595cb-njsxc\" (UID: \"d91d133a-9029-403e-9096-db48df37b7ff\") " pod="openshift-keda/keda-operator-ffbb595cb-njsxc" Apr 16 14:05:21.014584 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:21.014563 2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-7c9f485588-k7lfz"] Apr 16 14:05:21.020732 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:21.020705 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrh57\" (UniqueName: \"kubernetes.io/projected/d91d133a-9029-403e-9096-db48df37b7ff-kube-api-access-jrh57\") pod \"keda-operator-ffbb595cb-njsxc\" (UID: \"d91d133a-9029-403e-9096-db48df37b7ff\") " pod="openshift-keda/keda-operator-ffbb595cb-njsxc" Apr 16 14:05:21.107318 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:21.107286 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj8dr\" (UniqueName: \"kubernetes.io/projected/5e23a0a8-f51e-4ba2-b129-375f568ed228-kube-api-access-mj8dr\") pod \"keda-metrics-apiserver-7c9f485588-k7lfz\" (UID: \"5e23a0a8-f51e-4ba2-b129-375f568ed228\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-k7lfz" Apr 16 14:05:21.107318 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:21.107320 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/5e23a0a8-f51e-4ba2-b129-375f568ed228-temp-vol\") pod \"keda-metrics-apiserver-7c9f485588-k7lfz\" (UID: \"5e23a0a8-f51e-4ba2-b129-375f568ed228\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-k7lfz" Apr 16 14:05:21.107501 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:21.107340 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5e23a0a8-f51e-4ba2-b129-375f568ed228-certificates\") pod \"keda-metrics-apiserver-7c9f485588-k7lfz\" (UID: \"5e23a0a8-f51e-4ba2-b129-375f568ed228\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-k7lfz" Apr 16 14:05:21.208685 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:21.208650 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mj8dr\" (UniqueName: \"kubernetes.io/projected/5e23a0a8-f51e-4ba2-b129-375f568ed228-kube-api-access-mj8dr\") pod \"keda-metrics-apiserver-7c9f485588-k7lfz\" (UID: \"5e23a0a8-f51e-4ba2-b129-375f568ed228\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-k7lfz" Apr 16 14:05:21.208685 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:21.208688 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/5e23a0a8-f51e-4ba2-b129-375f568ed228-temp-vol\") pod \"keda-metrics-apiserver-7c9f485588-k7lfz\" (UID: \"5e23a0a8-f51e-4ba2-b129-375f568ed228\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-k7lfz" Apr 16 14:05:21.208885 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:21.208792 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5e23a0a8-f51e-4ba2-b129-375f568ed228-certificates\") pod \"keda-metrics-apiserver-7c9f485588-k7lfz\" (UID: \"5e23a0a8-f51e-4ba2-b129-375f568ed228\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-k7lfz" Apr 16 14:05:21.208925 ip-10-0-136-114 kubenswrapper[2574]: E0416 14:05:21.208884 2574 secret.go:281] references non-existent secret key: tls.crt Apr 16 14:05:21.208925 ip-10-0-136-114 kubenswrapper[2574]: E0416 14:05:21.208896 2574 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt Apr 16 14:05:21.208925 ip-10-0-136-114 kubenswrapper[2574]: E0416 14:05:21.208911 2574 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-7c9f485588-k7lfz: references non-existent secret key: tls.crt Apr 16 14:05:21.209025 ip-10-0-136-114 kubenswrapper[2574]: E0416 14:05:21.208957 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5e23a0a8-f51e-4ba2-b129-375f568ed228-certificates podName:5e23a0a8-f51e-4ba2-b129-375f568ed228 nodeName:}" failed. No retries permitted until 2026-04-16 14:05:21.708943154 +0000 UTC m=+491.538678002 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/5e23a0a8-f51e-4ba2-b129-375f568ed228-certificates") pod "keda-metrics-apiserver-7c9f485588-k7lfz" (UID: "5e23a0a8-f51e-4ba2-b129-375f568ed228") : references non-existent secret key: tls.crt Apr 16 14:05:21.209025 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:21.208989 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/5e23a0a8-f51e-4ba2-b129-375f568ed228-temp-vol\") pod \"keda-metrics-apiserver-7c9f485588-k7lfz\" (UID: \"5e23a0a8-f51e-4ba2-b129-375f568ed228\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-k7lfz" Apr 16 14:05:21.219413 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:21.219387 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-5q7r9" event={"ID":"180d28e1-662d-4b44-88d5-05d25814bb46","Type":"ContainerStarted","Data":"2cc9afbaad8edb3cf6fe82ba75c8162f3b0e6e4b4b124308fa0e03074396c4fc"} Apr 16 14:05:21.219547 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:21.219434 2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-5q7r9" Apr 16 14:05:21.229604 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:21.229570 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj8dr\" (UniqueName: \"kubernetes.io/projected/5e23a0a8-f51e-4ba2-b129-375f568ed228-kube-api-access-mj8dr\") pod \"keda-metrics-apiserver-7c9f485588-k7lfz\" (UID: \"5e23a0a8-f51e-4ba2-b129-375f568ed228\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-k7lfz" Apr 16 14:05:21.277040 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:21.276900 2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-5q7r9" podStartSLOduration=2.17555906 podStartE2EDuration="6.276885116s" podCreationTimestamp="2026-04-16 14:05:15 +0000 UTC" firstStartedPulling="2026-04-16 14:05:16.115398507 +0000 UTC m=+485.945133355" lastFinishedPulling="2026-04-16 14:05:20.216724563 +0000 UTC m=+490.046459411" observedRunningTime="2026-04-16 14:05:21.274264898 +0000 UTC m=+491.103999764" watchObservedRunningTime="2026-04-16 14:05:21.276885116 +0000 UTC m=+491.106619987" Apr 16 14:05:21.511182 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:21.511141 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d91d133a-9029-403e-9096-db48df37b7ff-certificates\") pod \"keda-operator-ffbb595cb-njsxc\" (UID: \"d91d133a-9029-403e-9096-db48df37b7ff\") " pod="openshift-keda/keda-operator-ffbb595cb-njsxc" Apr 16 14:05:21.511326 ip-10-0-136-114 kubenswrapper[2574]: E0416 14:05:21.511278 2574 secret.go:281] references non-existent secret key: ca.crt Apr 16 14:05:21.511326 ip-10-0-136-114 kubenswrapper[2574]: E0416 14:05:21.511297 2574 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt Apr 16 14:05:21.511326 ip-10-0-136-114 kubenswrapper[2574]: E0416 14:05:21.511305 2574 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-ffbb595cb-njsxc: references non-existent secret key: ca.crt Apr 16 14:05:21.511461 ip-10-0-136-114 kubenswrapper[2574]: E0416 14:05:21.511361 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d91d133a-9029-403e-9096-db48df37b7ff-certificates podName:d91d133a-9029-403e-9096-db48df37b7ff nodeName:}" failed. No retries permitted until 2026-04-16 14:05:22.511346348 +0000 UTC m=+492.341081196 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/d91d133a-9029-403e-9096-db48df37b7ff-certificates") pod "keda-operator-ffbb595cb-njsxc" (UID: "d91d133a-9029-403e-9096-db48df37b7ff") : references non-existent secret key: ca.crt Apr 16 14:05:21.713084 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:21.713053 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5e23a0a8-f51e-4ba2-b129-375f568ed228-certificates\") pod \"keda-metrics-apiserver-7c9f485588-k7lfz\" (UID: \"5e23a0a8-f51e-4ba2-b129-375f568ed228\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-k7lfz" Apr 16 14:05:21.713226 ip-10-0-136-114 kubenswrapper[2574]: E0416 14:05:21.713174 2574 secret.go:281] references non-existent secret key: tls.crt Apr 16 14:05:21.713226 ip-10-0-136-114 kubenswrapper[2574]: E0416 14:05:21.713187 2574 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt Apr 16 14:05:21.713226 ip-10-0-136-114 kubenswrapper[2574]: E0416 14:05:21.713203 2574 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-7c9f485588-k7lfz: references non-existent secret key: tls.crt Apr 16 14:05:21.713351 ip-10-0-136-114 kubenswrapper[2574]: E0416 14:05:21.713250 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5e23a0a8-f51e-4ba2-b129-375f568ed228-certificates podName:5e23a0a8-f51e-4ba2-b129-375f568ed228 nodeName:}" failed. No retries permitted until 2026-04-16 14:05:22.713237545 +0000 UTC m=+492.542972394 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/5e23a0a8-f51e-4ba2-b129-375f568ed228-certificates") pod "keda-metrics-apiserver-7c9f485588-k7lfz" (UID: "5e23a0a8-f51e-4ba2-b129-375f568ed228") : references non-existent secret key: tls.crt Apr 16 14:05:22.520129 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:22.520075 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d91d133a-9029-403e-9096-db48df37b7ff-certificates\") pod \"keda-operator-ffbb595cb-njsxc\" (UID: \"d91d133a-9029-403e-9096-db48df37b7ff\") " pod="openshift-keda/keda-operator-ffbb595cb-njsxc" Apr 16 14:05:22.520547 ip-10-0-136-114 kubenswrapper[2574]: E0416 14:05:22.520217 2574 secret.go:281] references non-existent secret key: ca.crt Apr 16 14:05:22.520547 ip-10-0-136-114 kubenswrapper[2574]: E0416 14:05:22.520239 2574 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt Apr 16 14:05:22.520547 ip-10-0-136-114 kubenswrapper[2574]: E0416 14:05:22.520249 2574 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-ffbb595cb-njsxc: references non-existent secret key: ca.crt Apr 16 14:05:22.520547 ip-10-0-136-114 kubenswrapper[2574]: E0416 14:05:22.520299 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d91d133a-9029-403e-9096-db48df37b7ff-certificates podName:d91d133a-9029-403e-9096-db48df37b7ff nodeName:}" failed. No retries permitted until 2026-04-16 14:05:24.520285067 +0000 UTC m=+494.350019914 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/d91d133a-9029-403e-9096-db48df37b7ff-certificates") pod "keda-operator-ffbb595cb-njsxc" (UID: "d91d133a-9029-403e-9096-db48df37b7ff") : references non-existent secret key: ca.crt Apr 16 14:05:22.722105 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:22.722066 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5e23a0a8-f51e-4ba2-b129-375f568ed228-certificates\") pod \"keda-metrics-apiserver-7c9f485588-k7lfz\" (UID: \"5e23a0a8-f51e-4ba2-b129-375f568ed228\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-k7lfz" Apr 16 14:05:22.722265 ip-10-0-136-114 kubenswrapper[2574]: E0416 14:05:22.722209 2574 secret.go:281] references non-existent secret key: tls.crt Apr 16 14:05:22.722265 ip-10-0-136-114 kubenswrapper[2574]: E0416 14:05:22.722228 2574 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt Apr 16 14:05:22.722265 ip-10-0-136-114 kubenswrapper[2574]: E0416 14:05:22.722247 2574 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-7c9f485588-k7lfz: references non-existent secret key: tls.crt Apr 16 14:05:22.722365 ip-10-0-136-114 kubenswrapper[2574]: E0416 14:05:22.722302 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5e23a0a8-f51e-4ba2-b129-375f568ed228-certificates podName:5e23a0a8-f51e-4ba2-b129-375f568ed228 nodeName:}" failed. No retries permitted until 2026-04-16 14:05:24.722288249 +0000 UTC m=+494.552023097 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/5e23a0a8-f51e-4ba2-b129-375f568ed228-certificates") pod "keda-metrics-apiserver-7c9f485588-k7lfz" (UID: "5e23a0a8-f51e-4ba2-b129-375f568ed228") : references non-existent secret key: tls.crt Apr 16 14:05:24.537554 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:24.537519 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d91d133a-9029-403e-9096-db48df37b7ff-certificates\") pod \"keda-operator-ffbb595cb-njsxc\" (UID: \"d91d133a-9029-403e-9096-db48df37b7ff\") " pod="openshift-keda/keda-operator-ffbb595cb-njsxc" Apr 16 14:05:24.537952 ip-10-0-136-114 kubenswrapper[2574]: E0416 14:05:24.537637 2574 secret.go:281] references non-existent secret key: ca.crt Apr 16 14:05:24.537952 ip-10-0-136-114 kubenswrapper[2574]: E0416 14:05:24.537649 2574 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt Apr 16 14:05:24.537952 ip-10-0-136-114 kubenswrapper[2574]: E0416 14:05:24.537664 2574 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-ffbb595cb-njsxc: references non-existent secret key: ca.crt Apr 16 14:05:24.537952 ip-10-0-136-114 kubenswrapper[2574]: E0416 14:05:24.537707 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d91d133a-9029-403e-9096-db48df37b7ff-certificates podName:d91d133a-9029-403e-9096-db48df37b7ff nodeName:}" failed. No retries permitted until 2026-04-16 14:05:28.537695784 +0000 UTC m=+498.367430632 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/d91d133a-9029-403e-9096-db48df37b7ff-certificates") pod "keda-operator-ffbb595cb-njsxc" (UID: "d91d133a-9029-403e-9096-db48df37b7ff") : references non-existent secret key: ca.crt Apr 16 14:05:24.739469 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:24.739435 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5e23a0a8-f51e-4ba2-b129-375f568ed228-certificates\") pod \"keda-metrics-apiserver-7c9f485588-k7lfz\" (UID: \"5e23a0a8-f51e-4ba2-b129-375f568ed228\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-k7lfz" Apr 16 14:05:24.739605 ip-10-0-136-114 kubenswrapper[2574]: E0416 14:05:24.739591 2574 secret.go:281] references non-existent secret key: tls.crt Apr 16 14:05:24.739665 ip-10-0-136-114 kubenswrapper[2574]: E0416 14:05:24.739613 2574 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt Apr 16 14:05:24.739665 ip-10-0-136-114 kubenswrapper[2574]: E0416 14:05:24.739635 2574 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-7c9f485588-k7lfz: references non-existent secret key: tls.crt Apr 16 14:05:24.739725 ip-10-0-136-114 kubenswrapper[2574]: E0416 14:05:24.739697 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5e23a0a8-f51e-4ba2-b129-375f568ed228-certificates podName:5e23a0a8-f51e-4ba2-b129-375f568ed228 nodeName:}" failed. No retries permitted until 2026-04-16 14:05:28.739679526 +0000 UTC m=+498.569414383 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/5e23a0a8-f51e-4ba2-b129-375f568ed228-certificates") pod "keda-metrics-apiserver-7c9f485588-k7lfz" (UID: "5e23a0a8-f51e-4ba2-b129-375f568ed228") : references non-existent secret key: tls.crt Apr 16 14:05:28.568834 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:28.568800 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d91d133a-9029-403e-9096-db48df37b7ff-certificates\") pod \"keda-operator-ffbb595cb-njsxc\" (UID: \"d91d133a-9029-403e-9096-db48df37b7ff\") " pod="openshift-keda/keda-operator-ffbb595cb-njsxc" Apr 16 14:05:28.571266 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:28.571243 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d91d133a-9029-403e-9096-db48df37b7ff-certificates\") pod \"keda-operator-ffbb595cb-njsxc\" (UID: \"d91d133a-9029-403e-9096-db48df37b7ff\") " pod="openshift-keda/keda-operator-ffbb595cb-njsxc" Apr 16 14:05:28.770628 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:28.770594 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5e23a0a8-f51e-4ba2-b129-375f568ed228-certificates\") pod \"keda-metrics-apiserver-7c9f485588-k7lfz\" (UID: \"5e23a0a8-f51e-4ba2-b129-375f568ed228\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-k7lfz" Apr 16 14:05:28.773038 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:28.773018 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5e23a0a8-f51e-4ba2-b129-375f568ed228-certificates\") pod \"keda-metrics-apiserver-7c9f485588-k7lfz\" (UID: \"5e23a0a8-f51e-4ba2-b129-375f568ed228\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-k7lfz" Apr 16 14:05:28.813268 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:28.813243 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-k7lfz" Apr 16 14:05:28.841886 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:28.841849 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-ffbb595cb-njsxc" Apr 16 14:05:28.943901 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:28.943876 2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-7c9f485588-k7lfz"] Apr 16 14:05:28.946998 ip-10-0-136-114 kubenswrapper[2574]: W0416 14:05:28.946970 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e23a0a8_f51e_4ba2_b129_375f568ed228.slice/crio-dc7f7a39499d9bf3094da0dbef71d902aad902217d67a3604ce4bd9c4ff1c6ee WatchSource:0}: Error finding container dc7f7a39499d9bf3094da0dbef71d902aad902217d67a3604ce4bd9c4ff1c6ee: Status 404 returned error can't find the container with id dc7f7a39499d9bf3094da0dbef71d902aad902217d67a3604ce4bd9c4ff1c6ee Apr 16 14:05:28.965443 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:28.965422 2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-ffbb595cb-njsxc"] Apr 16 14:05:28.967504 ip-10-0-136-114 kubenswrapper[2574]: W0416 14:05:28.967483 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd91d133a_9029_403e_9096_db48df37b7ff.slice/crio-7129362ac2a9d31c06bc49b8a8c54fd26a2312be0232c4332e79fd25886a7794 WatchSource:0}: Error finding container 7129362ac2a9d31c06bc49b8a8c54fd26a2312be0232c4332e79fd25886a7794: Status 404 returned error can't find the container with id 7129362ac2a9d31c06bc49b8a8c54fd26a2312be0232c4332e79fd25886a7794 Apr 16 14:05:29.245566 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:29.245533 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-ffbb595cb-njsxc" event={"ID":"d91d133a-9029-403e-9096-db48df37b7ff","Type":"ContainerStarted","Data":"7129362ac2a9d31c06bc49b8a8c54fd26a2312be0232c4332e79fd25886a7794"} Apr 16 14:05:29.246553 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:29.246529 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-k7lfz" event={"ID":"5e23a0a8-f51e-4ba2-b129-375f568ed228","Type":"ContainerStarted","Data":"dc7f7a39499d9bf3094da0dbef71d902aad902217d67a3604ce4bd9c4ff1c6ee"} Apr 16 14:05:33.263864 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:33.263771 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-ffbb595cb-njsxc" event={"ID":"d91d133a-9029-403e-9096-db48df37b7ff","Type":"ContainerStarted","Data":"1bfdcc55795ab6ae2ad19c6e0dac12475f05e6fd9c31a051e245abe6637bbc93"} Apr 16 14:05:33.264312 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:33.263913 2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-ffbb595cb-njsxc" Apr 16 14:05:33.265245 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:33.265221 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-k7lfz" event={"ID":"5e23a0a8-f51e-4ba2-b129-375f568ed228","Type":"ContainerStarted","Data":"8d8e97ba4cb7d40c53ef11d282d76a6c5c918ee6cdb66f9b286a08ed1f3f24a5"} Apr 16 14:05:33.265334 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:33.265316 2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-k7lfz" Apr 16 14:05:33.281561 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:33.281518 2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/keda-operator-ffbb595cb-njsxc" podStartSLOduration=9.369607344 podStartE2EDuration="13.281503939s" podCreationTimestamp="2026-04-16 14:05:20 +0000 UTC" firstStartedPulling="2026-04-16 14:05:28.968795593 +0000 UTC m=+498.798530441" lastFinishedPulling="2026-04-16 14:05:32.880692174 +0000 UTC m=+502.710427036" observedRunningTime="2026-04-16 14:05:33.279830262 +0000 UTC m=+503.109565157" watchObservedRunningTime="2026-04-16 14:05:33.281503939 +0000 UTC m=+503.111238809" Apr 16 14:05:33.300629 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:33.300585 2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-k7lfz" podStartSLOduration=9.373501842 podStartE2EDuration="13.300572783s" podCreationTimestamp="2026-04-16 14:05:20 +0000 UTC" firstStartedPulling="2026-04-16 14:05:28.948347776 +0000 UTC m=+498.778082628" lastFinishedPulling="2026-04-16 14:05:32.875418717 +0000 UTC m=+502.705153569" observedRunningTime="2026-04-16 14:05:33.298264016 +0000 UTC m=+503.127998887" watchObservedRunningTime="2026-04-16 14:05:33.300572783 +0000 UTC m=+503.130307652" Apr 16 14:05:42.224521 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:42.224490 2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-5q7r9" Apr 16 14:05:44.273557 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:44.273526 2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-k7lfz" Apr 16 14:05:54.271146 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:05:54.271067 2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-ffbb595cb-njsxc" Apr 16 14:06:27.832402 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:06:27.832364 2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-86cc847c5c-dvcw6"] Apr 16 14:06:27.834776 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:06:27.834754 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-86cc847c5c-dvcw6" Apr 16 14:06:27.837194 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:06:27.837170 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\"" Apr 16 14:06:27.837194 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:06:27.837171 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" Apr 16 14:06:27.837341 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:06:27.837245 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-m5pt5\"" Apr 16 14:06:27.837972 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:06:27.837958 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" Apr 16 14:06:27.846977 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:06:27.846957 2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-86cc847c5c-dvcw6"] Apr 16 14:06:27.858933 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:06:27.858912 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a5b029a4-c9ae-44ee-9e45-17637daf2c49-data\") pod \"seaweedfs-86cc847c5c-dvcw6\" (UID: \"a5b029a4-c9ae-44ee-9e45-17637daf2c49\") " pod="kserve/seaweedfs-86cc847c5c-dvcw6" Apr 16 14:06:27.859053 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:06:27.858957 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgz79\" (UniqueName: \"kubernetes.io/projected/a5b029a4-c9ae-44ee-9e45-17637daf2c49-kube-api-access-qgz79\") pod \"seaweedfs-86cc847c5c-dvcw6\" (UID: \"a5b029a4-c9ae-44ee-9e45-17637daf2c49\") " pod="kserve/seaweedfs-86cc847c5c-dvcw6" Apr 16 14:06:27.959575 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:06:27.959544 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qgz79\" (UniqueName: \"kubernetes.io/projected/a5b029a4-c9ae-44ee-9e45-17637daf2c49-kube-api-access-qgz79\") pod \"seaweedfs-86cc847c5c-dvcw6\" (UID: \"a5b029a4-c9ae-44ee-9e45-17637daf2c49\") " pod="kserve/seaweedfs-86cc847c5c-dvcw6" Apr 16 14:06:27.959730 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:06:27.959630 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a5b029a4-c9ae-44ee-9e45-17637daf2c49-data\") pod \"seaweedfs-86cc847c5c-dvcw6\" (UID: \"a5b029a4-c9ae-44ee-9e45-17637daf2c49\") " pod="kserve/seaweedfs-86cc847c5c-dvcw6" Apr 16 14:06:27.959949 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:06:27.959934 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a5b029a4-c9ae-44ee-9e45-17637daf2c49-data\") pod \"seaweedfs-86cc847c5c-dvcw6\" (UID: \"a5b029a4-c9ae-44ee-9e45-17637daf2c49\") " pod="kserve/seaweedfs-86cc847c5c-dvcw6" Apr 16 14:06:27.967872 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:06:27.967850 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgz79\" (UniqueName: \"kubernetes.io/projected/a5b029a4-c9ae-44ee-9e45-17637daf2c49-kube-api-access-qgz79\") pod \"seaweedfs-86cc847c5c-dvcw6\" (UID: \"a5b029a4-c9ae-44ee-9e45-17637daf2c49\") " pod="kserve/seaweedfs-86cc847c5c-dvcw6" Apr 16 14:06:28.144301 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:06:28.144264 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-86cc847c5c-dvcw6" Apr 16 14:06:28.264842 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:06:28.264731 2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-86cc847c5c-dvcw6"] Apr 16 14:06:28.267256 ip-10-0-136-114 kubenswrapper[2574]: W0416 14:06:28.267230 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5b029a4_c9ae_44ee_9e45_17637daf2c49.slice/crio-595780697f86c8dd8f2fdbf9dccc9957cfb15415084dcba51bf060ca06dd492e WatchSource:0}: Error finding container 595780697f86c8dd8f2fdbf9dccc9957cfb15415084dcba51bf060ca06dd492e: Status 404 returned error can't find the container with id 595780697f86c8dd8f2fdbf9dccc9957cfb15415084dcba51bf060ca06dd492e Apr 16 14:06:28.441252 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:06:28.441166 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-86cc847c5c-dvcw6" event={"ID":"a5b029a4-c9ae-44ee-9e45-17637daf2c49","Type":"ContainerStarted","Data":"595780697f86c8dd8f2fdbf9dccc9957cfb15415084dcba51bf060ca06dd492e"} Apr 16 14:06:31.455562 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:06:31.455481 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-86cc847c5c-dvcw6" event={"ID":"a5b029a4-c9ae-44ee-9e45-17637daf2c49","Type":"ContainerStarted","Data":"7d05ffa80d6dd77fb2920294431ff45e3a26718af6cf02fef7e87563c415beb6"} Apr 16 14:06:31.455919 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:06:31.455605 2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-86cc847c5c-dvcw6" Apr 16 14:06:31.482115 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:06:31.482069 2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/seaweedfs-86cc847c5c-dvcw6" podStartSLOduration=1.589763417 podStartE2EDuration="4.48205472s" podCreationTimestamp="2026-04-16 14:06:27 +0000 UTC" firstStartedPulling="2026-04-16 14:06:28.268469644 +0000 UTC m=+558.098204495" lastFinishedPulling="2026-04-16 14:06:31.160760949 +0000 UTC m=+560.990495798" observedRunningTime="2026-04-16 14:06:31.480389411 +0000 UTC m=+561.310124303" watchObservedRunningTime="2026-04-16 14:06:31.48205472 +0000 UTC m=+561.311789591" Apr 16 14:06:37.461084 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:06:37.461056 2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-86cc847c5c-dvcw6" Apr 16 14:06:58.820851 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:06:58.820813 2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-d57fbdccb-fzglm"] Apr 16 14:06:58.823323 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:06:58.823299 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d57fbdccb-fzglm" Apr 16 14:06:58.836715 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:06:58.836690 2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d57fbdccb-fzglm"] Apr 16 14:06:58.914053 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:06:58.914015 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/62f18cc1-2bd2-4106-a06b-98a99acf67f3-oauth-serving-cert\") pod \"console-d57fbdccb-fzglm\" (UID: \"62f18cc1-2bd2-4106-a06b-98a99acf67f3\") " pod="openshift-console/console-d57fbdccb-fzglm" Apr 16 14:06:58.914229 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:06:58.914177 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/62f18cc1-2bd2-4106-a06b-98a99acf67f3-console-config\") pod \"console-d57fbdccb-fzglm\" (UID: \"62f18cc1-2bd2-4106-a06b-98a99acf67f3\") " pod="openshift-console/console-d57fbdccb-fzglm" Apr 16 14:06:58.914229 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:06:58.914206 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psrtz\" (UniqueName: \"kubernetes.io/projected/62f18cc1-2bd2-4106-a06b-98a99acf67f3-kube-api-access-psrtz\") pod \"console-d57fbdccb-fzglm\" (UID: \"62f18cc1-2bd2-4106-a06b-98a99acf67f3\") " pod="openshift-console/console-d57fbdccb-fzglm" Apr 16 14:06:58.914396 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:06:58.914372 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/62f18cc1-2bd2-4106-a06b-98a99acf67f3-service-ca\") pod \"console-d57fbdccb-fzglm\" (UID: \"62f18cc1-2bd2-4106-a06b-98a99acf67f3\") " pod="openshift-console/console-d57fbdccb-fzglm" Apr 16 14:06:58.914491 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:06:58.914473 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62f18cc1-2bd2-4106-a06b-98a99acf67f3-trusted-ca-bundle\") pod \"console-d57fbdccb-fzglm\" (UID: \"62f18cc1-2bd2-4106-a06b-98a99acf67f3\") " pod="openshift-console/console-d57fbdccb-fzglm" Apr 16 14:06:58.914563 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:06:58.914547 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/62f18cc1-2bd2-4106-a06b-98a99acf67f3-console-serving-cert\") pod \"console-d57fbdccb-fzglm\" (UID: \"62f18cc1-2bd2-4106-a06b-98a99acf67f3\") " pod="openshift-console/console-d57fbdccb-fzglm" Apr 16 14:06:58.914624 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:06:58.914577 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/62f18cc1-2bd2-4106-a06b-98a99acf67f3-console-oauth-config\") pod \"console-d57fbdccb-fzglm\" (UID: \"62f18cc1-2bd2-4106-a06b-98a99acf67f3\") " pod="openshift-console/console-d57fbdccb-fzglm" Apr 16 14:06:59.016039 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:06:59.015995 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62f18cc1-2bd2-4106-a06b-98a99acf67f3-trusted-ca-bundle\") pod \"console-d57fbdccb-fzglm\" (UID: \"62f18cc1-2bd2-4106-a06b-98a99acf67f3\") " pod="openshift-console/console-d57fbdccb-fzglm" Apr 16 14:06:59.016209 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:06:59.016056 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/62f18cc1-2bd2-4106-a06b-98a99acf67f3-console-serving-cert\") pod \"console-d57fbdccb-fzglm\" (UID: \"62f18cc1-2bd2-4106-a06b-98a99acf67f3\") " pod="openshift-console/console-d57fbdccb-fzglm" Apr 16 14:06:59.016209 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:06:59.016074 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/62f18cc1-2bd2-4106-a06b-98a99acf67f3-console-oauth-config\") pod \"console-d57fbdccb-fzglm\" (UID: \"62f18cc1-2bd2-4106-a06b-98a99acf67f3\") " pod="openshift-console/console-d57fbdccb-fzglm" Apr 16 14:06:59.016209 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:06:59.016102 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/62f18cc1-2bd2-4106-a06b-98a99acf67f3-oauth-serving-cert\") pod \"console-d57fbdccb-fzglm\" (UID: \"62f18cc1-2bd2-4106-a06b-98a99acf67f3\") " pod="openshift-console/console-d57fbdccb-fzglm" Apr 16 14:06:59.016209 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:06:59.016130 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/62f18cc1-2bd2-4106-a06b-98a99acf67f3-console-config\") pod \"console-d57fbdccb-fzglm\" (UID: \"62f18cc1-2bd2-4106-a06b-98a99acf67f3\") " pod="openshift-console/console-d57fbdccb-fzglm" Apr 16 14:06:59.016209 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:06:59.016146 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-psrtz\" (UniqueName: \"kubernetes.io/projected/62f18cc1-2bd2-4106-a06b-98a99acf67f3-kube-api-access-psrtz\") pod \"console-d57fbdccb-fzglm\" (UID: \"62f18cc1-2bd2-4106-a06b-98a99acf67f3\") " pod="openshift-console/console-d57fbdccb-fzglm" Apr 16 14:06:59.016209 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:06:59.016172 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/62f18cc1-2bd2-4106-a06b-98a99acf67f3-service-ca\") pod \"console-d57fbdccb-fzglm\" (UID: \"62f18cc1-2bd2-4106-a06b-98a99acf67f3\") " pod="openshift-console/console-d57fbdccb-fzglm" Apr 16 14:06:59.016935 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:06:59.016907 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/62f18cc1-2bd2-4106-a06b-98a99acf67f3-console-config\") pod \"console-d57fbdccb-fzglm\" (UID: \"62f18cc1-2bd2-4106-a06b-98a99acf67f3\") " pod="openshift-console/console-d57fbdccb-fzglm" Apr 16 14:06:59.016935 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:06:59.016932 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/62f18cc1-2bd2-4106-a06b-98a99acf67f3-service-ca\") pod \"console-d57fbdccb-fzglm\" (UID: \"62f18cc1-2bd2-4106-a06b-98a99acf67f3\") " pod="openshift-console/console-d57fbdccb-fzglm" Apr 16 14:06:59.017106 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:06:59.016907 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/62f18cc1-2bd2-4106-a06b-98a99acf67f3-oauth-serving-cert\") pod \"console-d57fbdccb-fzglm\" (UID: \"62f18cc1-2bd2-4106-a06b-98a99acf67f3\") " pod="openshift-console/console-d57fbdccb-fzglm" Apr 16 14:06:59.017106 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:06:59.016993 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62f18cc1-2bd2-4106-a06b-98a99acf67f3-trusted-ca-bundle\") pod \"console-d57fbdccb-fzglm\" (UID: \"62f18cc1-2bd2-4106-a06b-98a99acf67f3\") " pod="openshift-console/console-d57fbdccb-fzglm" Apr 16 14:06:59.018621 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:06:59.018597 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/62f18cc1-2bd2-4106-a06b-98a99acf67f3-console-serving-cert\") pod \"console-d57fbdccb-fzglm\" (UID: \"62f18cc1-2bd2-4106-a06b-98a99acf67f3\") " pod="openshift-console/console-d57fbdccb-fzglm" Apr 16 14:06:59.018764 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:06:59.018707 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/62f18cc1-2bd2-4106-a06b-98a99acf67f3-console-oauth-config\") pod \"console-d57fbdccb-fzglm\" (UID: \"62f18cc1-2bd2-4106-a06b-98a99acf67f3\") " pod="openshift-console/console-d57fbdccb-fzglm" Apr 16 14:06:59.025009 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:06:59.024989 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psrtz\" (UniqueName: \"kubernetes.io/projected/62f18cc1-2bd2-4106-a06b-98a99acf67f3-kube-api-access-psrtz\") pod \"console-d57fbdccb-fzglm\" (UID: \"62f18cc1-2bd2-4106-a06b-98a99acf67f3\") " pod="openshift-console/console-d57fbdccb-fzglm" Apr 16 14:06:59.133052 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:06:59.133010 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d57fbdccb-fzglm" Apr 16 14:06:59.260261 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:06:59.260163 2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d57fbdccb-fzglm"] Apr 16 14:06:59.263090 ip-10-0-136-114 kubenswrapper[2574]: W0416 14:06:59.263059 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62f18cc1_2bd2_4106_a06b_98a99acf67f3.slice/crio-1e83eac1d1b2d821ca065ecc0e37b6736bd11757219d2d23b2279cf579a99b30 WatchSource:0}: Error finding container 1e83eac1d1b2d821ca065ecc0e37b6736bd11757219d2d23b2279cf579a99b30: Status 404 returned error can't find the container with id 1e83eac1d1b2d821ca065ecc0e37b6736bd11757219d2d23b2279cf579a99b30 Apr 16 14:06:59.547559 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:06:59.547471 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d57fbdccb-fzglm" event={"ID":"62f18cc1-2bd2-4106-a06b-98a99acf67f3","Type":"ContainerStarted","Data":"b2495ce2494eafdc6a8523f1d3ca1f1129ba3c8d07d0b5b9e2999098e6168421"} Apr 16 14:06:59.547559 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:06:59.547511 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d57fbdccb-fzglm" event={"ID":"62f18cc1-2bd2-4106-a06b-98a99acf67f3","Type":"ContainerStarted","Data":"1e83eac1d1b2d821ca065ecc0e37b6736bd11757219d2d23b2279cf579a99b30"} Apr 16 14:06:59.567771 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:06:59.567709 2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-d57fbdccb-fzglm" podStartSLOduration=1.567693698 podStartE2EDuration="1.567693698s" podCreationTimestamp="2026-04-16 14:06:58 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 14:06:59.565896273 +0000 UTC m=+589.395631164" watchObservedRunningTime="2026-04-16 14:06:59.567693698 +0000 UTC m=+589.397428569" Apr 16 14:07:09.133896 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:09.133846 2574 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-d57fbdccb-fzglm" Apr 16 14:07:09.133896 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:09.133894 2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-d57fbdccb-fzglm" Apr 16 14:07:09.138653 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:09.138630 2574 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-d57fbdccb-fzglm" Apr 16 14:07:09.586510 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:09.586481 2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-d57fbdccb-fzglm" Apr 16 14:07:09.635630 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:09.635598 2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-695dd5755c-lkc7t"] Apr 16 14:07:10.699118 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:10.699093 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6667474d89-tj5v6_5b9fe9ed-1a79-4590-ad6e-74acb8e02aed/cluster-monitoring-operator/0.log" Apr 16 14:07:10.700229 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:10.700209 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6667474d89-tj5v6_5b9fe9ed-1a79-4590-ad6e-74acb8e02aed/cluster-monitoring-operator/0.log" Apr 16 14:07:34.659661 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:34.659606 2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-695dd5755c-lkc7t" podUID="68515d67-48e0-4d3f-bc23-bfe7fbb9cc63" containerName="console" containerID="cri-o://c2cc5ab609627f9e810a8ff49a6b486adb0aaf7ab9fcf9351bcbd6e27855395d" gracePeriod=15 Apr 16 14:07:34.903870 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:34.903848 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-695dd5755c-lkc7t_68515d67-48e0-4d3f-bc23-bfe7fbb9cc63/console/0.log" Apr 16 14:07:34.903992 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:34.903907 2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-695dd5755c-lkc7t" Apr 16 14:07:35.013081 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:35.013014 2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68515d67-48e0-4d3f-bc23-bfe7fbb9cc63-trusted-ca-bundle\") pod \"68515d67-48e0-4d3f-bc23-bfe7fbb9cc63\" (UID: \"68515d67-48e0-4d3f-bc23-bfe7fbb9cc63\") " Apr 16 14:07:35.013081 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:35.013058 2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/68515d67-48e0-4d3f-bc23-bfe7fbb9cc63-service-ca\") pod \"68515d67-48e0-4d3f-bc23-bfe7fbb9cc63\" (UID: \"68515d67-48e0-4d3f-bc23-bfe7fbb9cc63\") " Apr 16 14:07:35.013232 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:35.013117 2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/68515d67-48e0-4d3f-bc23-bfe7fbb9cc63-console-config\") pod \"68515d67-48e0-4d3f-bc23-bfe7fbb9cc63\" (UID: \"68515d67-48e0-4d3f-bc23-bfe7fbb9cc63\") " Apr 16 14:07:35.013232 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:35.013151 2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/68515d67-48e0-4d3f-bc23-bfe7fbb9cc63-console-oauth-config\") pod \"68515d67-48e0-4d3f-bc23-bfe7fbb9cc63\" (UID: \"68515d67-48e0-4d3f-bc23-bfe7fbb9cc63\") " Apr 16 14:07:35.013314 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:35.013235 2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65m7x\" (UniqueName: \"kubernetes.io/projected/68515d67-48e0-4d3f-bc23-bfe7fbb9cc63-kube-api-access-65m7x\") pod \"68515d67-48e0-4d3f-bc23-bfe7fbb9cc63\" (UID: \"68515d67-48e0-4d3f-bc23-bfe7fbb9cc63\") " Apr 16 14:07:35.013314 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:35.013262 2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/68515d67-48e0-4d3f-bc23-bfe7fbb9cc63-console-serving-cert\") pod \"68515d67-48e0-4d3f-bc23-bfe7fbb9cc63\" (UID: \"68515d67-48e0-4d3f-bc23-bfe7fbb9cc63\") " Apr 16 14:07:35.013314 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:35.013296 2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/68515d67-48e0-4d3f-bc23-bfe7fbb9cc63-oauth-serving-cert\") pod \"68515d67-48e0-4d3f-bc23-bfe7fbb9cc63\" (UID: \"68515d67-48e0-4d3f-bc23-bfe7fbb9cc63\") " Apr 16 14:07:35.013468 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:35.013405 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68515d67-48e0-4d3f-bc23-bfe7fbb9cc63-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "68515d67-48e0-4d3f-bc23-bfe7fbb9cc63" (UID: "68515d67-48e0-4d3f-bc23-bfe7fbb9cc63"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 16 14:07:35.013546 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:35.013520 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68515d67-48e0-4d3f-bc23-bfe7fbb9cc63-service-ca" (OuterVolumeSpecName: "service-ca") pod "68515d67-48e0-4d3f-bc23-bfe7fbb9cc63" (UID: "68515d67-48e0-4d3f-bc23-bfe7fbb9cc63"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 16 14:07:35.013605 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:35.013540 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68515d67-48e0-4d3f-bc23-bfe7fbb9cc63-console-config" (OuterVolumeSpecName: "console-config") pod "68515d67-48e0-4d3f-bc23-bfe7fbb9cc63" (UID: "68515d67-48e0-4d3f-bc23-bfe7fbb9cc63"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 16 14:07:35.013706 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:35.013537 2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68515d67-48e0-4d3f-bc23-bfe7fbb9cc63-trusted-ca-bundle\") on node \"ip-10-0-136-114.ec2.internal\" DevicePath \"\"" Apr 16 14:07:35.013831 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:35.013737 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68515d67-48e0-4d3f-bc23-bfe7fbb9cc63-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "68515d67-48e0-4d3f-bc23-bfe7fbb9cc63" (UID: "68515d67-48e0-4d3f-bc23-bfe7fbb9cc63"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 16 14:07:35.015398 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:35.015374 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68515d67-48e0-4d3f-bc23-bfe7fbb9cc63-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "68515d67-48e0-4d3f-bc23-bfe7fbb9cc63" (UID: "68515d67-48e0-4d3f-bc23-bfe7fbb9cc63"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 16 14:07:35.015797 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:35.015772 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68515d67-48e0-4d3f-bc23-bfe7fbb9cc63-kube-api-access-65m7x" (OuterVolumeSpecName: "kube-api-access-65m7x") pod "68515d67-48e0-4d3f-bc23-bfe7fbb9cc63" (UID: "68515d67-48e0-4d3f-bc23-bfe7fbb9cc63"). InnerVolumeSpecName "kube-api-access-65m7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 16 14:07:35.015797 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:35.015771 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68515d67-48e0-4d3f-bc23-bfe7fbb9cc63-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "68515d67-48e0-4d3f-bc23-bfe7fbb9cc63" (UID: "68515d67-48e0-4d3f-bc23-bfe7fbb9cc63"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 16 14:07:35.114584 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:35.114560 2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/68515d67-48e0-4d3f-bc23-bfe7fbb9cc63-console-config\") on node \"ip-10-0-136-114.ec2.internal\" DevicePath \"\"" Apr 16 14:07:35.114584 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:35.114582 2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/68515d67-48e0-4d3f-bc23-bfe7fbb9cc63-console-oauth-config\") on node \"ip-10-0-136-114.ec2.internal\" DevicePath \"\"" Apr 16 14:07:35.114716 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:35.114592 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65m7x\" (UniqueName: \"kubernetes.io/projected/68515d67-48e0-4d3f-bc23-bfe7fbb9cc63-kube-api-access-65m7x\") on node \"ip-10-0-136-114.ec2.internal\" DevicePath \"\"" Apr 16 14:07:35.114716 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:35.114602 2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/68515d67-48e0-4d3f-bc23-bfe7fbb9cc63-console-serving-cert\") on node \"ip-10-0-136-114.ec2.internal\" DevicePath \"\"" Apr 16 14:07:35.114716 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:35.114612 2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/68515d67-48e0-4d3f-bc23-bfe7fbb9cc63-oauth-serving-cert\") on node \"ip-10-0-136-114.ec2.internal\" DevicePath \"\"" Apr 16 14:07:35.114716 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:35.114621 2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/68515d67-48e0-4d3f-bc23-bfe7fbb9cc63-service-ca\") on node \"ip-10-0-136-114.ec2.internal\" DevicePath \"\"" Apr 16 14:07:35.671206 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:35.671183 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-695dd5755c-lkc7t_68515d67-48e0-4d3f-bc23-bfe7fbb9cc63/console/0.log" Apr 16 14:07:35.671596 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:35.671221 2574 generic.go:358] "Generic (PLEG): container finished" podID="68515d67-48e0-4d3f-bc23-bfe7fbb9cc63" containerID="c2cc5ab609627f9e810a8ff49a6b486adb0aaf7ab9fcf9351bcbd6e27855395d" exitCode=2 Apr 16 14:07:35.671596 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:35.671255 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-695dd5755c-lkc7t" event={"ID":"68515d67-48e0-4d3f-bc23-bfe7fbb9cc63","Type":"ContainerDied","Data":"c2cc5ab609627f9e810a8ff49a6b486adb0aaf7ab9fcf9351bcbd6e27855395d"} Apr 16 14:07:35.671596 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:35.671276 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-695dd5755c-lkc7t" event={"ID":"68515d67-48e0-4d3f-bc23-bfe7fbb9cc63","Type":"ContainerDied","Data":"259700f9276db4d8a7c9431571d25de96ef072f526d439d00cc45198cfa7544b"} Apr 16 14:07:35.671596 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:35.671291 2574 scope.go:117] "RemoveContainer" containerID="c2cc5ab609627f9e810a8ff49a6b486adb0aaf7ab9fcf9351bcbd6e27855395d" Apr 16 14:07:35.671596 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:35.671303 2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-695dd5755c-lkc7t" Apr 16 14:07:35.680274 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:35.680255 2574 scope.go:117] "RemoveContainer" containerID="c2cc5ab609627f9e810a8ff49a6b486adb0aaf7ab9fcf9351bcbd6e27855395d" Apr 16 14:07:35.680483 ip-10-0-136-114 kubenswrapper[2574]: E0416 14:07:35.680466 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2cc5ab609627f9e810a8ff49a6b486adb0aaf7ab9fcf9351bcbd6e27855395d\": container with ID starting with c2cc5ab609627f9e810a8ff49a6b486adb0aaf7ab9fcf9351bcbd6e27855395d not found: ID does not exist" containerID="c2cc5ab609627f9e810a8ff49a6b486adb0aaf7ab9fcf9351bcbd6e27855395d" Apr 16 14:07:35.680525 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:35.680490 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2cc5ab609627f9e810a8ff49a6b486adb0aaf7ab9fcf9351bcbd6e27855395d"} err="failed to get container status \"c2cc5ab609627f9e810a8ff49a6b486adb0aaf7ab9fcf9351bcbd6e27855395d\": rpc error: code = NotFound desc = could not find container \"c2cc5ab609627f9e810a8ff49a6b486adb0aaf7ab9fcf9351bcbd6e27855395d\": container with ID starting with c2cc5ab609627f9e810a8ff49a6b486adb0aaf7ab9fcf9351bcbd6e27855395d not found: ID does not exist" Apr 16 14:07:35.692414 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:35.692392 2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-695dd5755c-lkc7t"] Apr 16 14:07:35.695610 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:35.695593 2574 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-695dd5755c-lkc7t"] Apr 16 14:07:35.878841 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:35.878813 2574 patch_prober.go:28] interesting pod/console-695dd5755c-lkc7t container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.133.0.25:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Apr 16 14:07:35.878944 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:35.878863 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/console-695dd5755c-lkc7t" podUID="68515d67-48e0-4d3f-bc23-bfe7fbb9cc63" containerName="console" probeResult="failure" output="Get \"https://10.133.0.25:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Apr 16 14:07:36.808707 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:36.808671 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68515d67-48e0-4d3f-bc23-bfe7fbb9cc63" path="/var/lib/kubelet/pods/68515d67-48e0-4d3f-bc23-bfe7fbb9cc63/volumes" Apr 16 14:07:38.324630 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:38.324595 2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-86f7b4b499-qkvb2"] Apr 16 14:07:38.325014 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:38.324973 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68515d67-48e0-4d3f-bc23-bfe7fbb9cc63" containerName="console" Apr 16 14:07:38.325014 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:38.324986 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="68515d67-48e0-4d3f-bc23-bfe7fbb9cc63" containerName="console" Apr 16 14:07:38.325087 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:38.325063 2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="68515d67-48e0-4d3f-bc23-bfe7fbb9cc63" containerName="console" Apr 16 14:07:38.328110 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:38.328092 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-86f7b4b499-qkvb2" Apr 16 14:07:38.330800 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:38.330777 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\"" Apr 16 14:07:38.330908 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:38.330832 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-qjz6x\"" Apr 16 14:07:38.337774 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:38.337724 2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-86f7b4b499-qkvb2"] Apr 16 14:07:38.341990 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:38.341965 2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-696fc77849-xmzbr"] Apr 16 14:07:38.344433 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:38.344414 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-696fc77849-xmzbr" Apr 16 14:07:38.346690 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:38.346670 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\"" Apr 16 14:07:38.346798 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:38.346707 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-r6f4n\"" Apr 16 14:07:38.354207 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:38.354186 2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-696fc77849-xmzbr"] Apr 16 14:07:38.447111 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:38.447083 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8c8144a5-5446-48ab-8815-9020cfc6de44-cert\") pod \"odh-model-controller-696fc77849-xmzbr\" (UID: \"8c8144a5-5446-48ab-8815-9020cfc6de44\") " pod="kserve/odh-model-controller-696fc77849-xmzbr" Apr 16 14:07:38.447253 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:38.447121 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebe717e9-86d8-40f6-b41e-8a1c44b7b982-tls-certs\") pod \"model-serving-api-86f7b4b499-qkvb2\" (UID: \"ebe717e9-86d8-40f6-b41e-8a1c44b7b982\") " pod="kserve/model-serving-api-86f7b4b499-qkvb2" Apr 16 14:07:38.447253 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:38.447162 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqp49\" (UniqueName: \"kubernetes.io/projected/ebe717e9-86d8-40f6-b41e-8a1c44b7b982-kube-api-access-kqp49\") pod \"model-serving-api-86f7b4b499-qkvb2\" (UID: \"ebe717e9-86d8-40f6-b41e-8a1c44b7b982\") " pod="kserve/model-serving-api-86f7b4b499-qkvb2" Apr 16 14:07:38.447253 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:38.447195 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk9ms\" (UniqueName: \"kubernetes.io/projected/8c8144a5-5446-48ab-8815-9020cfc6de44-kube-api-access-qk9ms\") pod \"odh-model-controller-696fc77849-xmzbr\" (UID: \"8c8144a5-5446-48ab-8815-9020cfc6de44\") " pod="kserve/odh-model-controller-696fc77849-xmzbr" Apr 16 14:07:38.548340 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:38.548307 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8c8144a5-5446-48ab-8815-9020cfc6de44-cert\") pod \"odh-model-controller-696fc77849-xmzbr\" (UID: \"8c8144a5-5446-48ab-8815-9020cfc6de44\") " pod="kserve/odh-model-controller-696fc77849-xmzbr" Apr 16 14:07:38.548495 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:38.548348 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebe717e9-86d8-40f6-b41e-8a1c44b7b982-tls-certs\") pod \"model-serving-api-86f7b4b499-qkvb2\" (UID: \"ebe717e9-86d8-40f6-b41e-8a1c44b7b982\") " pod="kserve/model-serving-api-86f7b4b499-qkvb2" Apr 16 14:07:38.548495 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:38.548382 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kqp49\" (UniqueName: \"kubernetes.io/projected/ebe717e9-86d8-40f6-b41e-8a1c44b7b982-kube-api-access-kqp49\") pod \"model-serving-api-86f7b4b499-qkvb2\" (UID: \"ebe717e9-86d8-40f6-b41e-8a1c44b7b982\") " pod="kserve/model-serving-api-86f7b4b499-qkvb2" Apr 16 14:07:38.548495 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:38.548419 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qk9ms\" (UniqueName: \"kubernetes.io/projected/8c8144a5-5446-48ab-8815-9020cfc6de44-kube-api-access-qk9ms\") pod \"odh-model-controller-696fc77849-xmzbr\" (UID: \"8c8144a5-5446-48ab-8815-9020cfc6de44\") " pod="kserve/odh-model-controller-696fc77849-xmzbr" Apr 16 14:07:38.548681 ip-10-0-136-114 kubenswrapper[2574]: E0416 14:07:38.548524 2574 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found Apr 16 14:07:38.548681 ip-10-0-136-114 kubenswrapper[2574]: E0416 14:07:38.548607 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebe717e9-86d8-40f6-b41e-8a1c44b7b982-tls-certs podName:ebe717e9-86d8-40f6-b41e-8a1c44b7b982 nodeName:}" failed. No retries permitted until 2026-04-16 14:07:39.048586564 +0000 UTC m=+628.878321412 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/ebe717e9-86d8-40f6-b41e-8a1c44b7b982-tls-certs") pod "model-serving-api-86f7b4b499-qkvb2" (UID: "ebe717e9-86d8-40f6-b41e-8a1c44b7b982") : secret "model-serving-api-tls" not found Apr 16 14:07:38.550668 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:38.550650 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8c8144a5-5446-48ab-8815-9020cfc6de44-cert\") pod \"odh-model-controller-696fc77849-xmzbr\" (UID: \"8c8144a5-5446-48ab-8815-9020cfc6de44\") " pod="kserve/odh-model-controller-696fc77849-xmzbr" Apr 16 14:07:38.559379 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:38.559354 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk9ms\" (UniqueName: \"kubernetes.io/projected/8c8144a5-5446-48ab-8815-9020cfc6de44-kube-api-access-qk9ms\") pod \"odh-model-controller-696fc77849-xmzbr\" (UID: \"8c8144a5-5446-48ab-8815-9020cfc6de44\") " pod="kserve/odh-model-controller-696fc77849-xmzbr" Apr 16 14:07:38.560015 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:38.559994 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqp49\" (UniqueName: \"kubernetes.io/projected/ebe717e9-86d8-40f6-b41e-8a1c44b7b982-kube-api-access-kqp49\") pod \"model-serving-api-86f7b4b499-qkvb2\" (UID: \"ebe717e9-86d8-40f6-b41e-8a1c44b7b982\") " pod="kserve/model-serving-api-86f7b4b499-qkvb2" Apr 16 14:07:38.655140 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:38.655094 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-696fc77849-xmzbr" Apr 16 14:07:38.777162 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:38.777104 2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-696fc77849-xmzbr"] Apr 16 14:07:38.780289 ip-10-0-136-114 kubenswrapper[2574]: W0416 14:07:38.780258 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c8144a5_5446_48ab_8815_9020cfc6de44.slice/crio-5bc11bdc852d48dfff161accab4f3d068e5ce7296e51236a76f4458cfe05c4f1 WatchSource:0}: Error finding container 5bc11bdc852d48dfff161accab4f3d068e5ce7296e51236a76f4458cfe05c4f1: Status 404 returned error can't find the container with id 5bc11bdc852d48dfff161accab4f3d068e5ce7296e51236a76f4458cfe05c4f1 Apr 16 14:07:39.053315 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:39.053236 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebe717e9-86d8-40f6-b41e-8a1c44b7b982-tls-certs\") pod \"model-serving-api-86f7b4b499-qkvb2\" (UID: \"ebe717e9-86d8-40f6-b41e-8a1c44b7b982\") " pod="kserve/model-serving-api-86f7b4b499-qkvb2" Apr 16 14:07:39.055466 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:39.055448 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebe717e9-86d8-40f6-b41e-8a1c44b7b982-tls-certs\") pod \"model-serving-api-86f7b4b499-qkvb2\" (UID: \"ebe717e9-86d8-40f6-b41e-8a1c44b7b982\") " pod="kserve/model-serving-api-86f7b4b499-qkvb2" Apr 16 14:07:39.239035 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:39.239005 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-86f7b4b499-qkvb2" Apr 16 14:07:39.354796 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:39.354771 2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-86f7b4b499-qkvb2"] Apr 16 14:07:39.357026 ip-10-0-136-114 kubenswrapper[2574]: W0416 14:07:39.356989 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebe717e9_86d8_40f6_b41e_8a1c44b7b982.slice/crio-d5c3fcf05074757211995082f7042370ab61fcf128b344a412f4155e275b89a9 WatchSource:0}: Error finding container d5c3fcf05074757211995082f7042370ab61fcf128b344a412f4155e275b89a9: Status 404 returned error can't find the container with id d5c3fcf05074757211995082f7042370ab61fcf128b344a412f4155e275b89a9 Apr 16 14:07:39.686889 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:39.686848 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-86f7b4b499-qkvb2" event={"ID":"ebe717e9-86d8-40f6-b41e-8a1c44b7b982","Type":"ContainerStarted","Data":"d5c3fcf05074757211995082f7042370ab61fcf128b344a412f4155e275b89a9"} Apr 16 14:07:39.687887 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:39.687863 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-696fc77849-xmzbr" event={"ID":"8c8144a5-5446-48ab-8815-9020cfc6de44","Type":"ContainerStarted","Data":"5bc11bdc852d48dfff161accab4f3d068e5ce7296e51236a76f4458cfe05c4f1"} Apr 16 14:07:43.708891 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:43.708850 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-696fc77849-xmzbr" event={"ID":"8c8144a5-5446-48ab-8815-9020cfc6de44","Type":"ContainerStarted","Data":"690b5aa262ca1ff75d4b7759532e812062e29c2e41ed358fd7c94af6c030cdec"} Apr 16 14:07:43.709375 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:43.709079 2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-696fc77849-xmzbr" Apr 16 14:07:43.710560 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:43.710536 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-86f7b4b499-qkvb2" event={"ID":"ebe717e9-86d8-40f6-b41e-8a1c44b7b982","Type":"ContainerStarted","Data":"52dffaa0d3d04cebeb43a8d4a4edcbdadc4f139e4bf4eb7b8b7292718364b2e4"} Apr 16 14:07:43.710705 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:43.710692 2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-86f7b4b499-qkvb2" Apr 16 14:07:43.772878 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:43.772834 2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/model-serving-api-86f7b4b499-qkvb2" podStartSLOduration=2.168717474 podStartE2EDuration="5.772821735s" podCreationTimestamp="2026-04-16 14:07:38 +0000 UTC" firstStartedPulling="2026-04-16 14:07:39.358772847 +0000 UTC m=+629.188507694" lastFinishedPulling="2026-04-16 14:07:42.962877104 +0000 UTC m=+632.792611955" observedRunningTime="2026-04-16 14:07:43.770308416 +0000 UTC m=+633.600043308" watchObservedRunningTime="2026-04-16 14:07:43.772821735 +0000 UTC m=+633.602556605" Apr 16 14:07:43.773014 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:43.772913 2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/odh-model-controller-696fc77849-xmzbr" podStartSLOduration=1.591661208 podStartE2EDuration="5.772909023s" podCreationTimestamp="2026-04-16 14:07:38 +0000 UTC" firstStartedPulling="2026-04-16 14:07:38.782038613 +0000 UTC m=+628.611773461" lastFinishedPulling="2026-04-16 14:07:42.963286429 +0000 UTC m=+632.793021276" observedRunningTime="2026-04-16 14:07:43.737113031 +0000 UTC m=+633.566847901" watchObservedRunningTime="2026-04-16 14:07:43.772909023 +0000 UTC m=+633.602643890" Apr 16 14:07:54.720102 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:54.720073 2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-696fc77849-xmzbr" Apr 16 14:07:54.722321 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:07:54.722300 2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-86f7b4b499-qkvb2" Apr 16 14:11:31.099301 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:11:31.099263 2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-bd7db-78d659848c-xmbs5"] Apr 16 14:11:31.102542 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:11:31.102521 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-bd7db-78d659848c-xmbs5" Apr 16 14:11:31.104914 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:11:31.104893 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-bd7db-kube-rbac-proxy-sar-config\"" Apr 16 14:11:31.105085 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:11:31.105073 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Apr 16 14:11:31.105128 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:11:31.105072 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9pgb7\"" Apr 16 14:11:31.105643 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:11:31.105628 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-bd7db-serving-cert\"" Apr 16 14:11:31.110359 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:11:31.110336 2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-bd7db-78d659848c-xmbs5"] Apr 16 14:11:31.182635 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:11:31.182585 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e543c7f7-1cf7-464f-b819-74fc9125e99e-proxy-tls\") pod \"model-chainer-raw-bd7db-78d659848c-xmbs5\" (UID: \"e543c7f7-1cf7-464f-b819-74fc9125e99e\") " pod="kserve-ci-e2e-test/model-chainer-raw-bd7db-78d659848c-xmbs5" Apr 16 14:11:31.182635 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:11:31.182631 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e543c7f7-1cf7-464f-b819-74fc9125e99e-openshift-service-ca-bundle\") pod \"model-chainer-raw-bd7db-78d659848c-xmbs5\" (UID: \"e543c7f7-1cf7-464f-b819-74fc9125e99e\") " pod="kserve-ci-e2e-test/model-chainer-raw-bd7db-78d659848c-xmbs5" Apr 16 14:11:31.283992 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:11:31.283942 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e543c7f7-1cf7-464f-b819-74fc9125e99e-proxy-tls\") pod \"model-chainer-raw-bd7db-78d659848c-xmbs5\" (UID: \"e543c7f7-1cf7-464f-b819-74fc9125e99e\") " pod="kserve-ci-e2e-test/model-chainer-raw-bd7db-78d659848c-xmbs5" Apr 16 14:11:31.283992 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:11:31.284002 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e543c7f7-1cf7-464f-b819-74fc9125e99e-openshift-service-ca-bundle\") pod \"model-chainer-raw-bd7db-78d659848c-xmbs5\" (UID: \"e543c7f7-1cf7-464f-b819-74fc9125e99e\") " pod="kserve-ci-e2e-test/model-chainer-raw-bd7db-78d659848c-xmbs5" Apr 16 14:11:31.284203 ip-10-0-136-114 kubenswrapper[2574]: E0416 14:11:31.284092 2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/model-chainer-raw-bd7db-serving-cert: secret "model-chainer-raw-bd7db-serving-cert" not found Apr 16 14:11:31.284203 ip-10-0-136-114 kubenswrapper[2574]: E0416 14:11:31.284157 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e543c7f7-1cf7-464f-b819-74fc9125e99e-proxy-tls podName:e543c7f7-1cf7-464f-b819-74fc9125e99e nodeName:}" failed. No retries permitted until 2026-04-16 14:11:31.784142224 +0000 UTC m=+861.613877072 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e543c7f7-1cf7-464f-b819-74fc9125e99e-proxy-tls") pod "model-chainer-raw-bd7db-78d659848c-xmbs5" (UID: "e543c7f7-1cf7-464f-b819-74fc9125e99e") : secret "model-chainer-raw-bd7db-serving-cert" not found Apr 16 14:11:31.284573 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:11:31.284556 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e543c7f7-1cf7-464f-b819-74fc9125e99e-openshift-service-ca-bundle\") pod \"model-chainer-raw-bd7db-78d659848c-xmbs5\" (UID: \"e543c7f7-1cf7-464f-b819-74fc9125e99e\") " pod="kserve-ci-e2e-test/model-chainer-raw-bd7db-78d659848c-xmbs5" Apr 16 14:11:31.789866 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:11:31.789839 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e543c7f7-1cf7-464f-b819-74fc9125e99e-proxy-tls\") pod \"model-chainer-raw-bd7db-78d659848c-xmbs5\" (UID: \"e543c7f7-1cf7-464f-b819-74fc9125e99e\") " pod="kserve-ci-e2e-test/model-chainer-raw-bd7db-78d659848c-xmbs5" Apr 16 14:11:31.792101 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:11:31.792069 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e543c7f7-1cf7-464f-b819-74fc9125e99e-proxy-tls\") pod \"model-chainer-raw-bd7db-78d659848c-xmbs5\" (UID: \"e543c7f7-1cf7-464f-b819-74fc9125e99e\") " pod="kserve-ci-e2e-test/model-chainer-raw-bd7db-78d659848c-xmbs5" Apr 16 14:11:32.013643 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:11:32.013614 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-bd7db-78d659848c-xmbs5" Apr 16 14:11:32.131874 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:11:32.131851 2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-bd7db-78d659848c-xmbs5"] Apr 16 14:11:32.134557 ip-10-0-136-114 kubenswrapper[2574]: W0416 14:11:32.134526 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode543c7f7_1cf7_464f_b819_74fc9125e99e.slice/crio-6e582aaee8e86c6f930d9f41fd3fdf8eb9128ff01172498198dd0c529d5a51b1 WatchSource:0}: Error finding container 6e582aaee8e86c6f930d9f41fd3fdf8eb9128ff01172498198dd0c529d5a51b1: Status 404 returned error can't find the container with id 6e582aaee8e86c6f930d9f41fd3fdf8eb9128ff01172498198dd0c529d5a51b1 Apr 16 14:11:32.136371 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:11:32.136353 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Apr 16 14:11:32.461668 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:11:32.461630 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-bd7db-78d659848c-xmbs5" event={"ID":"e543c7f7-1cf7-464f-b819-74fc9125e99e","Type":"ContainerStarted","Data":"6e582aaee8e86c6f930d9f41fd3fdf8eb9128ff01172498198dd0c529d5a51b1"} Apr 16 14:11:36.476311 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:11:36.476277 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-bd7db-78d659848c-xmbs5" event={"ID":"e543c7f7-1cf7-464f-b819-74fc9125e99e","Type":"ContainerStarted","Data":"bb46ea951498864ebbdac79830c0a71d974a95e50af9963f19f5662c0fd68d15"} Apr 16 14:11:36.476780 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:11:36.476362 2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-bd7db-78d659848c-xmbs5" Apr 16 14:11:36.506265 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:11:36.506214 2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-raw-bd7db-78d659848c-xmbs5" podStartSLOduration=1.743108394 podStartE2EDuration="5.506198013s" podCreationTimestamp="2026-04-16 14:11:31 +0000 UTC" firstStartedPulling="2026-04-16 14:11:32.136543503 +0000 UTC m=+861.966278361" lastFinishedPulling="2026-04-16 14:11:35.899633131 +0000 UTC m=+865.729367980" observedRunningTime="2026-04-16 14:11:36.505047958 +0000 UTC m=+866.334782828" watchObservedRunningTime="2026-04-16 14:11:36.506198013 +0000 UTC m=+866.335932885" Apr 16 14:11:41.154735 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:11:41.154690 2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-bd7db-78d659848c-xmbs5"] Apr 16 14:11:41.155194 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:11:41.155041 2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-raw-bd7db-78d659848c-xmbs5" podUID="e543c7f7-1cf7-464f-b819-74fc9125e99e" containerName="model-chainer-raw-bd7db" containerID="cri-o://bb46ea951498864ebbdac79830c0a71d974a95e50af9963f19f5662c0fd68d15" gracePeriod=30 Apr 16 14:11:41.160632 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:11:41.160603 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-bd7db-78d659848c-xmbs5" podUID="e543c7f7-1cf7-464f-b819-74fc9125e99e" containerName="model-chainer-raw-bd7db" probeResult="failure" output="HTTP probe failed with statuscode: 503" Apr 16 14:11:46.159648 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:11:46.159603 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-bd7db-78d659848c-xmbs5" podUID="e543c7f7-1cf7-464f-b819-74fc9125e99e" containerName="model-chainer-raw-bd7db" probeResult="failure" output="HTTP probe failed with statuscode: 503" Apr 16 14:11:51.158819 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:11:51.158782 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-bd7db-78d659848c-xmbs5" podUID="e543c7f7-1cf7-464f-b819-74fc9125e99e" containerName="model-chainer-raw-bd7db" probeResult="failure" output="HTTP probe failed with statuscode: 503" Apr 16 14:11:56.159236 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:11:56.159152 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-bd7db-78d659848c-xmbs5" podUID="e543c7f7-1cf7-464f-b819-74fc9125e99e" containerName="model-chainer-raw-bd7db" probeResult="failure" output="HTTP probe failed with statuscode: 503" Apr 16 14:12:01.158521 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:12:01.158481 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-bd7db-78d659848c-xmbs5" podUID="e543c7f7-1cf7-464f-b819-74fc9125e99e" containerName="model-chainer-raw-bd7db" probeResult="failure" output="HTTP probe failed with statuscode: 503" Apr 16 14:12:06.158843 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:12:06.158802 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-bd7db-78d659848c-xmbs5" podUID="e543c7f7-1cf7-464f-b819-74fc9125e99e" containerName="model-chainer-raw-bd7db" probeResult="failure" output="HTTP probe failed with statuscode: 503" Apr 16 14:12:10.722455 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:12:10.722428 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6667474d89-tj5v6_5b9fe9ed-1a79-4590-ad6e-74acb8e02aed/cluster-monitoring-operator/0.log" Apr 16 14:12:10.724931 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:12:10.724911 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6667474d89-tj5v6_5b9fe9ed-1a79-4590-ad6e-74acb8e02aed/cluster-monitoring-operator/0.log" Apr 16 14:12:11.160158 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:12:11.160115 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-bd7db-78d659848c-xmbs5" podUID="e543c7f7-1cf7-464f-b819-74fc9125e99e" containerName="model-chainer-raw-bd7db" probeResult="failure" output="HTTP probe failed with statuscode: 503" Apr 16 14:12:11.594258 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:12:11.594227 2574 generic.go:358] "Generic (PLEG): container finished" podID="e543c7f7-1cf7-464f-b819-74fc9125e99e" containerID="bb46ea951498864ebbdac79830c0a71d974a95e50af9963f19f5662c0fd68d15" exitCode=0 Apr 16 14:12:11.594407 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:12:11.594264 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-bd7db-78d659848c-xmbs5" event={"ID":"e543c7f7-1cf7-464f-b819-74fc9125e99e","Type":"ContainerDied","Data":"bb46ea951498864ebbdac79830c0a71d974a95e50af9963f19f5662c0fd68d15"} Apr 16 14:12:11.790284 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:12:11.790264 2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-bd7db-78d659848c-xmbs5" Apr 16 14:12:11.940694 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:12:11.940631 2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e543c7f7-1cf7-464f-b819-74fc9125e99e-openshift-service-ca-bundle\") pod \"e543c7f7-1cf7-464f-b819-74fc9125e99e\" (UID: \"e543c7f7-1cf7-464f-b819-74fc9125e99e\") " Apr 16 14:12:11.940694 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:12:11.940676 2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e543c7f7-1cf7-464f-b819-74fc9125e99e-proxy-tls\") pod \"e543c7f7-1cf7-464f-b819-74fc9125e99e\" (UID: \"e543c7f7-1cf7-464f-b819-74fc9125e99e\") " Apr 16 14:12:11.940994 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:12:11.940970 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e543c7f7-1cf7-464f-b819-74fc9125e99e-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "e543c7f7-1cf7-464f-b819-74fc9125e99e" (UID: "e543c7f7-1cf7-464f-b819-74fc9125e99e"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 16 14:12:11.942728 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:12:11.942701 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e543c7f7-1cf7-464f-b819-74fc9125e99e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e543c7f7-1cf7-464f-b819-74fc9125e99e" (UID: "e543c7f7-1cf7-464f-b819-74fc9125e99e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 16 14:12:12.042216 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:12:12.042185 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e543c7f7-1cf7-464f-b819-74fc9125e99e-openshift-service-ca-bundle\") on node \"ip-10-0-136-114.ec2.internal\" DevicePath \"\"" Apr 16 14:12:12.042216 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:12:12.042210 2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e543c7f7-1cf7-464f-b819-74fc9125e99e-proxy-tls\") on node \"ip-10-0-136-114.ec2.internal\" DevicePath \"\"" Apr 16 14:12:12.598397 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:12:12.598358 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-bd7db-78d659848c-xmbs5" event={"ID":"e543c7f7-1cf7-464f-b819-74fc9125e99e","Type":"ContainerDied","Data":"6e582aaee8e86c6f930d9f41fd3fdf8eb9128ff01172498198dd0c529d5a51b1"} Apr 16 14:12:12.598581 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:12:12.598411 2574 scope.go:117] "RemoveContainer" containerID="bb46ea951498864ebbdac79830c0a71d974a95e50af9963f19f5662c0fd68d15" Apr 16 14:12:12.598581 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:12:12.598377 2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-bd7db-78d659848c-xmbs5" Apr 16 14:12:12.621999 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:12:12.621975 2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-bd7db-78d659848c-xmbs5"] Apr 16 14:12:12.626017 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:12:12.625993 2574 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-bd7db-78d659848c-xmbs5"] Apr 16 14:12:12.809207 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:12:12.809183 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e543c7f7-1cf7-464f-b819-74fc9125e99e" path="/var/lib/kubelet/pods/e543c7f7-1cf7-464f-b819-74fc9125e99e/volumes" Apr 16 14:13:11.385756 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:13:11.385710 2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-1d91c-77dfd7db99-vjtfz"] Apr 16 14:13:11.386120 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:13:11.386105 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e543c7f7-1cf7-464f-b819-74fc9125e99e" containerName="model-chainer-raw-bd7db" Apr 16 14:13:11.386161 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:13:11.386122 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e543c7f7-1cf7-464f-b819-74fc9125e99e" containerName="model-chainer-raw-bd7db" Apr 16 14:13:11.386204 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:13:11.386193 2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="e543c7f7-1cf7-464f-b819-74fc9125e99e" containerName="model-chainer-raw-bd7db" Apr 16 14:13:11.388495 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:13:11.388477 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-1d91c-77dfd7db99-vjtfz" Apr 16 14:13:11.391282 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:13:11.391254 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9pgb7\"" Apr 16 14:13:11.391282 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:13:11.391264 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-hpa-1d91c-serving-cert\"" Apr 16 14:13:11.391573 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:13:11.391551 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-hpa-1d91c-kube-rbac-proxy-sar-config\"" Apr 16 14:13:11.392110 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:13:11.392089 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Apr 16 14:13:11.403981 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:13:11.403962 2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-1d91c-77dfd7db99-vjtfz"] Apr 16 14:13:11.529439 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:13:11.529400 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2069363-806e-4c2b-8007-cd4d5d84ddbc-openshift-service-ca-bundle\") pod \"model-chainer-raw-hpa-1d91c-77dfd7db99-vjtfz\" (UID: \"b2069363-806e-4c2b-8007-cd4d5d84ddbc\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-1d91c-77dfd7db99-vjtfz" Apr 16 14:13:11.529599 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:13:11.529478 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2069363-806e-4c2b-8007-cd4d5d84ddbc-proxy-tls\") pod \"model-chainer-raw-hpa-1d91c-77dfd7db99-vjtfz\" (UID: \"b2069363-806e-4c2b-8007-cd4d5d84ddbc\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-1d91c-77dfd7db99-vjtfz" Apr 16 14:13:11.630143 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:13:11.630112 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2069363-806e-4c2b-8007-cd4d5d84ddbc-openshift-service-ca-bundle\") pod \"model-chainer-raw-hpa-1d91c-77dfd7db99-vjtfz\" (UID: \"b2069363-806e-4c2b-8007-cd4d5d84ddbc\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-1d91c-77dfd7db99-vjtfz" Apr 16 14:13:11.630325 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:13:11.630176 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2069363-806e-4c2b-8007-cd4d5d84ddbc-proxy-tls\") pod \"model-chainer-raw-hpa-1d91c-77dfd7db99-vjtfz\" (UID: \"b2069363-806e-4c2b-8007-cd4d5d84ddbc\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-1d91c-77dfd7db99-vjtfz" Apr 16 14:13:11.630820 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:13:11.630798 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2069363-806e-4c2b-8007-cd4d5d84ddbc-openshift-service-ca-bundle\") pod \"model-chainer-raw-hpa-1d91c-77dfd7db99-vjtfz\" (UID: \"b2069363-806e-4c2b-8007-cd4d5d84ddbc\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-1d91c-77dfd7db99-vjtfz" Apr 16 14:13:11.632497 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:13:11.632473 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2069363-806e-4c2b-8007-cd4d5d84ddbc-proxy-tls\") pod \"model-chainer-raw-hpa-1d91c-77dfd7db99-vjtfz\" (UID: \"b2069363-806e-4c2b-8007-cd4d5d84ddbc\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-1d91c-77dfd7db99-vjtfz" Apr 16 14:13:11.699193 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:13:11.699113 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-1d91c-77dfd7db99-vjtfz" Apr 16 14:13:11.820251 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:13:11.820139 2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-1d91c-77dfd7db99-vjtfz"] Apr 16 14:13:11.822773 ip-10-0-136-114 kubenswrapper[2574]: W0416 14:13:11.822734 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2069363_806e_4c2b_8007_cd4d5d84ddbc.slice/crio-f574ef405c8e47b99b6c640a23e8fc26709ef7e6bc4739253713753584a92f42 WatchSource:0}: Error finding container f574ef405c8e47b99b6c640a23e8fc26709ef7e6bc4739253713753584a92f42: Status 404 returned error can't find the container with id f574ef405c8e47b99b6c640a23e8fc26709ef7e6bc4739253713753584a92f42 Apr 16 14:13:12.802329 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:13:12.802292 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-1d91c-77dfd7db99-vjtfz" event={"ID":"b2069363-806e-4c2b-8007-cd4d5d84ddbc","Type":"ContainerStarted","Data":"c92feb280c78d93b618114df5692d2ad5d0c9aed76e34b6fb4170b36cff613a0"} Apr 16 14:13:12.802329 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:13:12.802330 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-1d91c-77dfd7db99-vjtfz" event={"ID":"b2069363-806e-4c2b-8007-cd4d5d84ddbc","Type":"ContainerStarted","Data":"f574ef405c8e47b99b6c640a23e8fc26709ef7e6bc4739253713753584a92f42"} Apr 16 14:13:12.802874 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:13:12.802428 2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-1d91c-77dfd7db99-vjtfz" Apr 16 14:13:12.821558 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:13:12.821509 2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-1d91c-77dfd7db99-vjtfz" podStartSLOduration=1.821496154 podStartE2EDuration="1.821496154s" podCreationTimestamp="2026-04-16 14:13:11 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 14:13:12.819226184 +0000 UTC m=+962.648961045" watchObservedRunningTime="2026-04-16 14:13:12.821496154 +0000 UTC m=+962.651231024" Apr 16 14:13:18.811415 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:13:18.811383 2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-1d91c-77dfd7db99-vjtfz" Apr 16 14:13:21.447240 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:13:21.447204 2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-1d91c-77dfd7db99-vjtfz"] Apr 16 14:13:21.447659 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:13:21.447429 2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-1d91c-77dfd7db99-vjtfz" podUID="b2069363-806e-4c2b-8007-cd4d5d84ddbc" containerName="model-chainer-raw-hpa-1d91c" containerID="cri-o://c92feb280c78d93b618114df5692d2ad5d0c9aed76e34b6fb4170b36cff613a0" gracePeriod=30 Apr 16 14:13:21.620393 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:13:21.620357 2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-7cc0c-predictor-7f8fd4fc94-zk549"] Apr 16 14:13:21.622938 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:13:21.622916 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-7cc0c-predictor-7f8fd4fc94-zk549" Apr 16 14:13:21.631136 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:13:21.631108 2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-7cc0c-predictor-7f8fd4fc94-zk549"] Apr 16 14:13:21.632820 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:13:21.632790 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-7cc0c-predictor-7f8fd4fc94-zk549" Apr 16 14:13:21.758510 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:13:21.758474 2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-7cc0c-predictor-7f8fd4fc94-zk549"] Apr 16 14:13:21.762410 ip-10-0-136-114 kubenswrapper[2574]: W0416 14:13:21.762383 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a3ef8e4_3752_4d52_878d_05d2ff127b72.slice/crio-3f50bd2fbcc6edefe1f1dc98ae75ee470cae2b5b3481cfe9262b3b49552282cd WatchSource:0}: Error finding container 3f50bd2fbcc6edefe1f1dc98ae75ee470cae2b5b3481cfe9262b3b49552282cd: Status 404 returned error can't find the container with id 3f50bd2fbcc6edefe1f1dc98ae75ee470cae2b5b3481cfe9262b3b49552282cd Apr 16 14:13:21.832919 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:13:21.832892 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-7cc0c-predictor-7f8fd4fc94-zk549" event={"ID":"7a3ef8e4-3752-4d52-878d-05d2ff127b72","Type":"ContainerStarted","Data":"3f50bd2fbcc6edefe1f1dc98ae75ee470cae2b5b3481cfe9262b3b49552282cd"} Apr 16 14:13:23.809367 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:13:23.809330 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-1d91c-77dfd7db99-vjtfz" podUID="b2069363-806e-4c2b-8007-cd4d5d84ddbc" containerName="model-chainer-raw-hpa-1d91c" probeResult="failure" output="HTTP probe failed with statuscode: 503" Apr 16 14:13:23.843538 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:13:23.843507 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-7cc0c-predictor-7f8fd4fc94-zk549" event={"ID":"7a3ef8e4-3752-4d52-878d-05d2ff127b72","Type":"ContainerStarted","Data":"eafc5b3adb8be1850d40a28abb1b2f994263232b628e3b00570cd95d0d0162c6"} Apr 16 14:13:23.843681 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:13:23.843668 2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-7cc0c-predictor-7f8fd4fc94-zk549" Apr 16 14:13:23.845553 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:13:23.845530 2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-7cc0c-predictor-7f8fd4fc94-zk549" Apr 16 14:13:23.858059 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:13:23.857981 2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-7cc0c-predictor-7f8fd4fc94-zk549" podStartSLOduration=1.755618739 podStartE2EDuration="2.857967003s" podCreationTimestamp="2026-04-16 14:13:21 +0000 UTC" firstStartedPulling="2026-04-16 14:13:21.764177332 +0000 UTC m=+971.593912180" lastFinishedPulling="2026-04-16 14:13:22.866525597 +0000 UTC m=+972.696260444" observedRunningTime="2026-04-16 14:13:23.856672753 +0000 UTC m=+973.686407624" watchObservedRunningTime="2026-04-16 14:13:23.857967003 +0000 UTC m=+973.687701873" Apr 16 14:13:28.810471 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:13:28.810431 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-1d91c-77dfd7db99-vjtfz" podUID="b2069363-806e-4c2b-8007-cd4d5d84ddbc" containerName="model-chainer-raw-hpa-1d91c" probeResult="failure" output="HTTP probe failed with statuscode: 503" Apr 16 14:13:33.810055 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:13:33.810012 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-1d91c-77dfd7db99-vjtfz" podUID="b2069363-806e-4c2b-8007-cd4d5d84ddbc" containerName="model-chainer-raw-hpa-1d91c" probeResult="failure" output="HTTP probe failed with statuscode: 503" Apr 16 14:13:33.810441 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:13:33.810124 2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-1d91c-77dfd7db99-vjtfz" Apr 16 14:13:38.809917 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:13:38.809879 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-1d91c-77dfd7db99-vjtfz" podUID="b2069363-806e-4c2b-8007-cd4d5d84ddbc" containerName="model-chainer-raw-hpa-1d91c" probeResult="failure" output="HTTP probe failed with statuscode: 503" Apr 16 14:13:43.810080 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:13:43.810039 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-1d91c-77dfd7db99-vjtfz" podUID="b2069363-806e-4c2b-8007-cd4d5d84ddbc" containerName="model-chainer-raw-hpa-1d91c" probeResult="failure" output="HTTP probe failed with statuscode: 503" Apr 16 14:13:48.809559 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:13:48.809522 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-1d91c-77dfd7db99-vjtfz" podUID="b2069363-806e-4c2b-8007-cd4d5d84ddbc" containerName="model-chainer-raw-hpa-1d91c" probeResult="failure" output="HTTP probe failed with statuscode: 503" Apr 16 14:13:51.936750 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:13:51.936716 2574 generic.go:358] "Generic (PLEG): container finished" podID="b2069363-806e-4c2b-8007-cd4d5d84ddbc" containerID="c92feb280c78d93b618114df5692d2ad5d0c9aed76e34b6fb4170b36cff613a0" exitCode=0 Apr 16 14:13:51.937173 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:13:51.936791 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-1d91c-77dfd7db99-vjtfz" event={"ID":"b2069363-806e-4c2b-8007-cd4d5d84ddbc","Type":"ContainerDied","Data":"c92feb280c78d93b618114df5692d2ad5d0c9aed76e34b6fb4170b36cff613a0"} Apr 16 14:13:52.094883 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:13:52.094860 2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-1d91c-77dfd7db99-vjtfz" Apr 16 14:13:52.149026 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:13:52.148994 2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2069363-806e-4c2b-8007-cd4d5d84ddbc-openshift-service-ca-bundle\") pod \"b2069363-806e-4c2b-8007-cd4d5d84ddbc\" (UID: \"b2069363-806e-4c2b-8007-cd4d5d84ddbc\") " Apr 16 14:13:52.149026 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:13:52.149026 2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2069363-806e-4c2b-8007-cd4d5d84ddbc-proxy-tls\") pod \"b2069363-806e-4c2b-8007-cd4d5d84ddbc\" (UID: \"b2069363-806e-4c2b-8007-cd4d5d84ddbc\") " Apr 16 14:13:52.149378 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:13:52.149352 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2069363-806e-4c2b-8007-cd4d5d84ddbc-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "b2069363-806e-4c2b-8007-cd4d5d84ddbc" (UID: "b2069363-806e-4c2b-8007-cd4d5d84ddbc"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 16 14:13:52.151053 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:13:52.151033 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2069363-806e-4c2b-8007-cd4d5d84ddbc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b2069363-806e-4c2b-8007-cd4d5d84ddbc" (UID: "b2069363-806e-4c2b-8007-cd4d5d84ddbc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 16 14:13:52.249940 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:13:52.249864 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2069363-806e-4c2b-8007-cd4d5d84ddbc-openshift-service-ca-bundle\") on node \"ip-10-0-136-114.ec2.internal\" DevicePath \"\"" Apr 16 14:13:52.249940 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:13:52.249890 2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2069363-806e-4c2b-8007-cd4d5d84ddbc-proxy-tls\") on node \"ip-10-0-136-114.ec2.internal\" DevicePath \"\"" Apr 16 14:13:52.941218 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:13:52.941182 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-1d91c-77dfd7db99-vjtfz" event={"ID":"b2069363-806e-4c2b-8007-cd4d5d84ddbc","Type":"ContainerDied","Data":"f574ef405c8e47b99b6c640a23e8fc26709ef7e6bc4739253713753584a92f42"} Apr 16 14:13:52.941218 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:13:52.941225 2574 scope.go:117] "RemoveContainer" containerID="c92feb280c78d93b618114df5692d2ad5d0c9aed76e34b6fb4170b36cff613a0" Apr 16 14:13:52.941682 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:13:52.941238 2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-1d91c-77dfd7db99-vjtfz" Apr 16 14:13:52.961048 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:13:52.961019 2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-1d91c-77dfd7db99-vjtfz"] Apr 16 14:13:52.967704 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:13:52.967685 2574 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-1d91c-77dfd7db99-vjtfz"] Apr 16 14:13:54.809242 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:13:54.809208 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2069363-806e-4c2b-8007-cd4d5d84ddbc" path="/var/lib/kubelet/pods/b2069363-806e-4c2b-8007-cd4d5d84ddbc/volumes" Apr 16 14:14:56.705328 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:14:56.705253 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-7cc0c-predictor-7f8fd4fc94-zk549_7a3ef8e4-3752-4d52-878d-05d2ff127b72/kserve-container/0.log" Apr 16 14:14:56.986389 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:14:56.986306 2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-7cc0c-predictor-7f8fd4fc94-zk549"] Apr 16 14:14:56.986544 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:14:56.986523 2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-7cc0c-predictor-7f8fd4fc94-zk549" podUID="7a3ef8e4-3752-4d52-878d-05d2ff127b72" containerName="kserve-container" containerID="cri-o://eafc5b3adb8be1850d40a28abb1b2f994263232b628e3b00570cd95d0d0162c6" gracePeriod=30 Apr 16 14:14:57.157947 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:14:57.157917 2574 generic.go:358] "Generic (PLEG): container finished" podID="7a3ef8e4-3752-4d52-878d-05d2ff127b72" containerID="eafc5b3adb8be1850d40a28abb1b2f994263232b628e3b00570cd95d0d0162c6" exitCode=2 Apr 16 14:14:57.158092 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:14:57.157982 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-7cc0c-predictor-7f8fd4fc94-zk549" event={"ID":"7a3ef8e4-3752-4d52-878d-05d2ff127b72","Type":"ContainerDied","Data":"eafc5b3adb8be1850d40a28abb1b2f994263232b628e3b00570cd95d0d0162c6"} Apr 16 14:14:57.234677 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:14:57.234656 2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-7cc0c-predictor-7f8fd4fc94-zk549" Apr 16 14:14:58.162138 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:14:58.162108 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-7cc0c-predictor-7f8fd4fc94-zk549" event={"ID":"7a3ef8e4-3752-4d52-878d-05d2ff127b72","Type":"ContainerDied","Data":"3f50bd2fbcc6edefe1f1dc98ae75ee470cae2b5b3481cfe9262b3b49552282cd"} Apr 16 14:14:58.162563 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:14:58.162147 2574 scope.go:117] "RemoveContainer" containerID="eafc5b3adb8be1850d40a28abb1b2f994263232b628e3b00570cd95d0d0162c6" Apr 16 14:14:58.162563 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:14:58.162152 2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-7cc0c-predictor-7f8fd4fc94-zk549" Apr 16 14:14:58.182945 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:14:58.182917 2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-7cc0c-predictor-7f8fd4fc94-zk549"] Apr 16 14:14:58.186306 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:14:58.186282 2574 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-7cc0c-predictor-7f8fd4fc94-zk549"] Apr 16 14:14:58.809221 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:14:58.809189 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a3ef8e4-3752-4d52-878d-05d2ff127b72" path="/var/lib/kubelet/pods/7a3ef8e4-3752-4d52-878d-05d2ff127b72/volumes" Apr 16 14:17:10.747957 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:17:10.747919 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6667474d89-tj5v6_5b9fe9ed-1a79-4590-ad6e-74acb8e02aed/cluster-monitoring-operator/0.log" Apr 16 14:17:10.750910 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:17:10.750889 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6667474d89-tj5v6_5b9fe9ed-1a79-4590-ad6e-74acb8e02aed/cluster-monitoring-operator/0.log" Apr 16 14:22:10.770681 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:10.770641 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6667474d89-tj5v6_5b9fe9ed-1a79-4590-ad6e-74acb8e02aed/cluster-monitoring-operator/0.log" Apr 16 14:22:10.774909 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:10.774888 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6667474d89-tj5v6_5b9fe9ed-1a79-4590-ad6e-74acb8e02aed/cluster-monitoring-operator/0.log" Apr 16 14:22:23.221779 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:23.221692 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-lk2d2_2ac404a2-5e6f-4a47-bc40-f8f182cd968e/global-pull-secret-syncer/0.log" Apr 16 14:22:23.327731 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:23.327702 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-sflxw_cab34005-67cd-4db7-80d0-1d5b72338051/konnectivity-agent/0.log" Apr 16 14:22:23.372558 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:23.372518 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-114.ec2.internal_031b74a3d41a1b92a3331318a0772b2d/haproxy/0.log" Apr 16 14:22:27.112348 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:27.112317 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_59f76eb3-ff88-4e98-9015-4a91c304f5ec/alertmanager/0.log" Apr 16 14:22:27.138542 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:27.138514 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_59f76eb3-ff88-4e98-9015-4a91c304f5ec/config-reloader/0.log" Apr 16 14:22:27.168629 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:27.168604 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_59f76eb3-ff88-4e98-9015-4a91c304f5ec/kube-rbac-proxy-web/0.log" Apr 16 14:22:27.192648 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:27.192630 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_59f76eb3-ff88-4e98-9015-4a91c304f5ec/kube-rbac-proxy/0.log" Apr 16 14:22:27.217272 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:27.217239 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_59f76eb3-ff88-4e98-9015-4a91c304f5ec/kube-rbac-proxy-metric/0.log" Apr 16 14:22:27.242031 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:27.241979 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_59f76eb3-ff88-4e98-9015-4a91c304f5ec/prom-label-proxy/0.log" Apr 16 14:22:27.267106 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:27.267089 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_59f76eb3-ff88-4e98-9015-4a91c304f5ec/init-config-reloader/0.log" Apr 16 14:22:27.306608 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:27.306589 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6667474d89-tj5v6_5b9fe9ed-1a79-4590-ad6e-74acb8e02aed/cluster-monitoring-operator/1.log" Apr 16 14:22:27.381255 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:27.381226 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6667474d89-tj5v6_5b9fe9ed-1a79-4590-ad6e-74acb8e02aed/cluster-monitoring-operator/0.log" Apr 16 14:22:27.406972 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:27.406904 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7479c89684-8xh94_74619757-38ff-48ef-9c0c-129c39811a4d/kube-state-metrics/0.log" Apr 16 14:22:27.431589 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:27.431569 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7479c89684-8xh94_74619757-38ff-48ef-9c0c-129c39811a4d/kube-rbac-proxy-main/0.log" Apr 16 14:22:27.454830 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:27.454806 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7479c89684-8xh94_74619757-38ff-48ef-9c0c-129c39811a4d/kube-rbac-proxy-self/0.log" Apr 16 14:22:27.632044 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:27.632017 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h7mfj_35436df9-1548-4b04-9e65-ca60b80486e7/node-exporter/0.log" Apr 16 14:22:27.655151 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:27.655128 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h7mfj_35436df9-1548-4b04-9e65-ca60b80486e7/kube-rbac-proxy/0.log" Apr 16 14:22:27.680364 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:27.680341 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h7mfj_35436df9-1548-4b04-9e65-ca60b80486e7/init-textfile/0.log" Apr 16 14:22:27.797203 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:27.797175 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-5669946b84-q57gd_0d969860-3250-40b5-b124-a01d0c99a8a1/kube-rbac-proxy-main/0.log" Apr 16 14:22:27.819836 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:27.819814 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-5669946b84-q57gd_0d969860-3250-40b5-b124-a01d0c99a8a1/kube-rbac-proxy-self/0.log" Apr 16 14:22:27.845130 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:27.845109 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-5669946b84-q57gd_0d969860-3250-40b5-b124-a01d0c99a8a1/openshift-state-metrics/0.log" Apr 16 14:22:28.137185 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:28.137157 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-9cb97cd87-cnfb5_4f118646-7573-4184-880d-eb313a7b063d/prometheus-operator-admission-webhook/0.log" Apr 16 14:22:28.178762 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:28.178725 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-64476d7dbf-cmlpq_dacf88d2-d40d-4130-8f88-e060c7b4835c/telemeter-client/0.log" Apr 16 14:22:28.231257 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:28.231237 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-64476d7dbf-cmlpq_dacf88d2-d40d-4130-8f88-e060c7b4835c/reload/0.log" Apr 16 14:22:28.261659 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:28.261623 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-64476d7dbf-cmlpq_dacf88d2-d40d-4130-8f88-e060c7b4835c/kube-rbac-proxy/0.log" Apr 16 14:22:28.303372 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:28.303350 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c94669ccd-k5vvr_78e0f4b8-9966-439a-98af-e8d7a5253a42/thanos-query/0.log" Apr 16 14:22:28.349431 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:28.349410 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c94669ccd-k5vvr_78e0f4b8-9966-439a-98af-e8d7a5253a42/kube-rbac-proxy-web/0.log" Apr 16 14:22:28.398012 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:28.397954 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c94669ccd-k5vvr_78e0f4b8-9966-439a-98af-e8d7a5253a42/kube-rbac-proxy/0.log" Apr 16 14:22:28.438369 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:28.438349 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c94669ccd-k5vvr_78e0f4b8-9966-439a-98af-e8d7a5253a42/prom-label-proxy/0.log" Apr 16 14:22:28.465913 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:28.465895 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c94669ccd-k5vvr_78e0f4b8-9966-439a-98af-e8d7a5253a42/kube-rbac-proxy-rules/0.log" Apr 16 14:22:28.489731 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:28.489711 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c94669ccd-k5vvr_78e0f4b8-9966-439a-98af-e8d7a5253a42/kube-rbac-proxy-metrics/0.log" Apr 16 14:22:30.297496 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:30.297458 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-d57fbdccb-fzglm_62f18cc1-2bd2-4106-a06b-98a99acf67f3/console/0.log" Apr 16 14:22:30.329562 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:30.329535 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-586b57c7b4-bd2j9_9b1f9023-4bfb-4ee6-9045-780848f02427/download-server/0.log" Apr 16 14:22:30.498645 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:30.498617 2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-s6bn5/perf-node-gather-daemonset-sl4j7"] Apr 16 14:22:30.498993 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:30.498980 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a3ef8e4-3752-4d52-878d-05d2ff127b72" containerName="kserve-container" Apr 16 14:22:30.499044 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:30.498995 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a3ef8e4-3752-4d52-878d-05d2ff127b72" containerName="kserve-container" Apr 16 14:22:30.499044 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:30.499005 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2069363-806e-4c2b-8007-cd4d5d84ddbc" containerName="model-chainer-raw-hpa-1d91c" Apr 16 14:22:30.499044 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:30.499012 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2069363-806e-4c2b-8007-cd4d5d84ddbc" containerName="model-chainer-raw-hpa-1d91c" Apr 16 14:22:30.499137 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:30.499070 2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="7a3ef8e4-3752-4d52-878d-05d2ff127b72" containerName="kserve-container" Apr 16 14:22:30.499137 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:30.499081 2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="b2069363-806e-4c2b-8007-cd4d5d84ddbc" containerName="model-chainer-raw-hpa-1d91c" Apr 16 14:22:30.501940 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:30.501926 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s6bn5/perf-node-gather-daemonset-sl4j7" Apr 16 14:22:30.504372 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:30.504353 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s6bn5\"/\"openshift-service-ca.crt\"" Apr 16 14:22:30.505331 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:30.505318 2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s6bn5\"/\"kube-root-ca.crt\"" Apr 16 14:22:30.505395 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:30.505306 2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-s6bn5\"/\"default-dockercfg-nnrnd\"" Apr 16 14:22:30.511068 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:30.511049 2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s6bn5/perf-node-gather-daemonset-sl4j7"] Apr 16 14:22:30.625778 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:30.625756 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/49bb8e9e-c50b-44ed-b8d5-6269ec9ba2fa-sys\") pod \"perf-node-gather-daemonset-sl4j7\" (UID: \"49bb8e9e-c50b-44ed-b8d5-6269ec9ba2fa\") " pod="openshift-must-gather-s6bn5/perf-node-gather-daemonset-sl4j7" Apr 16 14:22:30.625894 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:30.625791 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpp5w\" (UniqueName: \"kubernetes.io/projected/49bb8e9e-c50b-44ed-b8d5-6269ec9ba2fa-kube-api-access-vpp5w\") pod \"perf-node-gather-daemonset-sl4j7\" (UID: \"49bb8e9e-c50b-44ed-b8d5-6269ec9ba2fa\") " pod="openshift-must-gather-s6bn5/perf-node-gather-daemonset-sl4j7" Apr 16 14:22:30.625894 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:30.625840 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/49bb8e9e-c50b-44ed-b8d5-6269ec9ba2fa-lib-modules\") pod \"perf-node-gather-daemonset-sl4j7\" (UID: \"49bb8e9e-c50b-44ed-b8d5-6269ec9ba2fa\") " pod="openshift-must-gather-s6bn5/perf-node-gather-daemonset-sl4j7" Apr 16 14:22:30.625969 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:30.625893 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/49bb8e9e-c50b-44ed-b8d5-6269ec9ba2fa-proc\") pod \"perf-node-gather-daemonset-sl4j7\" (UID: \"49bb8e9e-c50b-44ed-b8d5-6269ec9ba2fa\") " pod="openshift-must-gather-s6bn5/perf-node-gather-daemonset-sl4j7" Apr 16 14:22:30.625969 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:30.625919 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/49bb8e9e-c50b-44ed-b8d5-6269ec9ba2fa-podres\") pod \"perf-node-gather-daemonset-sl4j7\" (UID: \"49bb8e9e-c50b-44ed-b8d5-6269ec9ba2fa\") " pod="openshift-must-gather-s6bn5/perf-node-gather-daemonset-sl4j7" Apr 16 14:22:30.726913 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:30.726884 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/49bb8e9e-c50b-44ed-b8d5-6269ec9ba2fa-proc\") pod \"perf-node-gather-daemonset-sl4j7\" (UID: \"49bb8e9e-c50b-44ed-b8d5-6269ec9ba2fa\") " pod="openshift-must-gather-s6bn5/perf-node-gather-daemonset-sl4j7" Apr 16 14:22:30.727052 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:30.726920 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/49bb8e9e-c50b-44ed-b8d5-6269ec9ba2fa-podres\") pod \"perf-node-gather-daemonset-sl4j7\" (UID: \"49bb8e9e-c50b-44ed-b8d5-6269ec9ba2fa\") " pod="openshift-must-gather-s6bn5/perf-node-gather-daemonset-sl4j7" Apr 16 14:22:30.727052 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:30.726977 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/49bb8e9e-c50b-44ed-b8d5-6269ec9ba2fa-sys\") pod \"perf-node-gather-daemonset-sl4j7\" (UID: \"49bb8e9e-c50b-44ed-b8d5-6269ec9ba2fa\") " pod="openshift-must-gather-s6bn5/perf-node-gather-daemonset-sl4j7" Apr 16 14:22:30.727052 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:30.727001 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vpp5w\" (UniqueName: \"kubernetes.io/projected/49bb8e9e-c50b-44ed-b8d5-6269ec9ba2fa-kube-api-access-vpp5w\") pod \"perf-node-gather-daemonset-sl4j7\" (UID: \"49bb8e9e-c50b-44ed-b8d5-6269ec9ba2fa\") " pod="openshift-must-gather-s6bn5/perf-node-gather-daemonset-sl4j7" Apr 16 14:22:30.727052 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:30.727014 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/49bb8e9e-c50b-44ed-b8d5-6269ec9ba2fa-proc\") pod \"perf-node-gather-daemonset-sl4j7\" (UID: \"49bb8e9e-c50b-44ed-b8d5-6269ec9ba2fa\") " pod="openshift-must-gather-s6bn5/perf-node-gather-daemonset-sl4j7" Apr 16 14:22:30.727052 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:30.727021 2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/49bb8e9e-c50b-44ed-b8d5-6269ec9ba2fa-lib-modules\") pod \"perf-node-gather-daemonset-sl4j7\" (UID: \"49bb8e9e-c50b-44ed-b8d5-6269ec9ba2fa\") " pod="openshift-must-gather-s6bn5/perf-node-gather-daemonset-sl4j7" Apr 16 14:22:30.727286 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:30.727074 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/49bb8e9e-c50b-44ed-b8d5-6269ec9ba2fa-sys\") pod \"perf-node-gather-daemonset-sl4j7\" (UID: \"49bb8e9e-c50b-44ed-b8d5-6269ec9ba2fa\") " pod="openshift-must-gather-s6bn5/perf-node-gather-daemonset-sl4j7" Apr 16 14:22:30.727286 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:30.727111 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/49bb8e9e-c50b-44ed-b8d5-6269ec9ba2fa-lib-modules\") pod \"perf-node-gather-daemonset-sl4j7\" (UID: \"49bb8e9e-c50b-44ed-b8d5-6269ec9ba2fa\") " pod="openshift-must-gather-s6bn5/perf-node-gather-daemonset-sl4j7" Apr 16 14:22:30.727286 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:30.727175 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/49bb8e9e-c50b-44ed-b8d5-6269ec9ba2fa-podres\") pod \"perf-node-gather-daemonset-sl4j7\" (UID: \"49bb8e9e-c50b-44ed-b8d5-6269ec9ba2fa\") " pod="openshift-must-gather-s6bn5/perf-node-gather-daemonset-sl4j7" Apr 16 14:22:30.734722 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:30.734695 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpp5w\" (UniqueName: \"kubernetes.io/projected/49bb8e9e-c50b-44ed-b8d5-6269ec9ba2fa-kube-api-access-vpp5w\") pod \"perf-node-gather-daemonset-sl4j7\" (UID: \"49bb8e9e-c50b-44ed-b8d5-6269ec9ba2fa\") " pod="openshift-must-gather-s6bn5/perf-node-gather-daemonset-sl4j7" Apr 16 14:22:30.813157 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:30.813135 2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s6bn5/perf-node-gather-daemonset-sl4j7" Apr 16 14:22:30.929102 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:30.929080 2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s6bn5/perf-node-gather-daemonset-sl4j7"] Apr 16 14:22:30.931179 ip-10-0-136-114 kubenswrapper[2574]: W0416 14:22:30.931154 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod49bb8e9e_c50b_44ed_b8d5_6269ec9ba2fa.slice/crio-79ee508835ec307baa7bd7c2236908ca4d604ebf4f3078234efeffd00635a6c5 WatchSource:0}: Error finding container 79ee508835ec307baa7bd7c2236908ca4d604ebf4f3078234efeffd00635a6c5: Status 404 returned error can't find the container with id 79ee508835ec307baa7bd7c2236908ca4d604ebf4f3078234efeffd00635a6c5 Apr 16 14:22:30.932677 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:30.932662 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Apr 16 14:22:31.490561 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:31.490536 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lnfdm_eded8e93-5cbc-435c-aab3-a909812ba63b/dns/0.log" Apr 16 14:22:31.514532 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:31.514507 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lnfdm_eded8e93-5cbc-435c-aab3-a909812ba63b/kube-rbac-proxy/0.log" Apr 16 14:22:31.590170 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:31.590146 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-x4qcz_c590b6a7-f5a5-476f-a0bd-fb8e8853b8cb/dns-node-resolver/0.log" Apr 16 14:22:31.648488 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:31.648464 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s6bn5/perf-node-gather-daemonset-sl4j7" event={"ID":"49bb8e9e-c50b-44ed-b8d5-6269ec9ba2fa","Type":"ContainerStarted","Data":"f33c2d189ce71fb67ece1c5f29db50c82115fa4488e60113f420168c2fe73e9b"} Apr 16 14:22:31.648622 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:31.648497 2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s6bn5/perf-node-gather-daemonset-sl4j7" event={"ID":"49bb8e9e-c50b-44ed-b8d5-6269ec9ba2fa","Type":"ContainerStarted","Data":"79ee508835ec307baa7bd7c2236908ca4d604ebf4f3078234efeffd00635a6c5"} Apr 16 14:22:31.648622 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:31.648600 2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-s6bn5/perf-node-gather-daemonset-sl4j7" Apr 16 14:22:31.665063 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:31.665025 2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-s6bn5/perf-node-gather-daemonset-sl4j7" podStartSLOduration=1.665013949 podStartE2EDuration="1.665013949s" podCreationTimestamp="2026-04-16 14:22:30 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 14:22:31.663429185 +0000 UTC m=+1521.493164055" watchObservedRunningTime="2026-04-16 14:22:31.665013949 +0000 UTC m=+1521.494748854" Apr 16 14:22:32.084139 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:32.084109 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-brqdj_48194990-0de3-45a9-acc5-6fcc4c2760d0/node-ca/0.log" Apr 16 14:22:32.796965 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:32.796936 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-69cb8777b6-kbr8q_e81d5797-d78f-46cf-b792-fdb668a624fc/router/0.log" Apr 16 14:22:33.178247 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:33.178213 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xjpgz_7ee1c9f7-5758-4734-a7b3-ea621cc6df95/serve-healthcheck-canary/0.log" Apr 16 14:22:33.715165 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:33.715135 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wdl6d_ff5a884a-7809-4fea-a804-78ce4afaa27e/kube-rbac-proxy/0.log" Apr 16 14:22:33.737104 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:33.737081 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wdl6d_ff5a884a-7809-4fea-a804-78ce4afaa27e/exporter/0.log" Apr 16 14:22:33.762806 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:33.762768 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wdl6d_ff5a884a-7809-4fea-a804-78ce4afaa27e/extractor/0.log" Apr 16 14:22:35.720958 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:35.720930 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-86f7b4b499-qkvb2_ebe717e9-86d8-40f6-b41e-8a1c44b7b982/server/0.log" Apr 16 14:22:35.800321 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:35.800296 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-696fc77849-xmzbr_8c8144a5-5446-48ab-8815-9020cfc6de44/manager/0.log" Apr 16 14:22:35.849456 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:35.849433 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-86cc847c5c-dvcw6_a5b029a4-c9ae-44ee-9e45-17637daf2c49/seaweedfs/0.log" Apr 16 14:22:37.660626 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:37.660592 2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-s6bn5/perf-node-gather-daemonset-sl4j7" Apr 16 14:22:41.263200 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:41.263172 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8js6c_5b83df07-d346-4caa-b358-294994a3d89b/kube-multus/0.log" Apr 16 14:22:41.487988 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:41.487963 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rnvkv_ce951cb8-753a-4902-be96-172949512289/kube-multus-additional-cni-plugins/0.log" Apr 16 14:22:41.512581 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:41.512557 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rnvkv_ce951cb8-753a-4902-be96-172949512289/egress-router-binary-copy/0.log" Apr 16 14:22:41.539258 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:41.539199 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rnvkv_ce951cb8-753a-4902-be96-172949512289/cni-plugins/0.log" Apr 16 14:22:41.564932 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:41.564909 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rnvkv_ce951cb8-753a-4902-be96-172949512289/bond-cni-plugin/0.log" Apr 16 14:22:41.591484 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:41.591461 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rnvkv_ce951cb8-753a-4902-be96-172949512289/routeoverride-cni/0.log" Apr 16 14:22:41.619312 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:41.619292 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rnvkv_ce951cb8-753a-4902-be96-172949512289/whereabouts-cni-bincopy/0.log" Apr 16 14:22:41.647932 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:41.647913 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rnvkv_ce951cb8-753a-4902-be96-172949512289/whereabouts-cni/0.log" Apr 16 14:22:41.906520 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:41.906485 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-5fh28_2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e/network-metrics-daemon/0.log" Apr 16 14:22:41.931469 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:41.931444 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-5fh28_2eb1b9a0-87a2-478e-a0ea-b2da8ceced2e/kube-rbac-proxy/0.log" Apr 16 14:22:42.777218 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:42.777190 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-99w5m_c18aa469-e518-4e2b-9d80-85e7e1d8ed94/ovn-controller/0.log" Apr 16 14:22:42.819519 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:42.819492 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-99w5m_c18aa469-e518-4e2b-9d80-85e7e1d8ed94/ovn-acl-logging/0.log" Apr 16 14:22:42.863590 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:42.863565 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-99w5m_c18aa469-e518-4e2b-9d80-85e7e1d8ed94/kube-rbac-proxy-node/0.log" Apr 16 14:22:42.907808 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:42.907784 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-99w5m_c18aa469-e518-4e2b-9d80-85e7e1d8ed94/kube-rbac-proxy-ovn-metrics/0.log" Apr 16 14:22:42.946802 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:42.946765 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-99w5m_c18aa469-e518-4e2b-9d80-85e7e1d8ed94/northd/0.log" Apr 16 14:22:42.993569 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:42.993547 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-99w5m_c18aa469-e518-4e2b-9d80-85e7e1d8ed94/nbdb/0.log" Apr 16 14:22:43.041610 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:43.041564 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-99w5m_c18aa469-e518-4e2b-9d80-85e7e1d8ed94/sbdb/0.log" Apr 16 14:22:43.153594 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:43.153571 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-99w5m_c18aa469-e518-4e2b-9d80-85e7e1d8ed94/ovnkube-controller/0.log" Apr 16 14:22:45.055869 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:45.055843 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-h4r6r_df2f75cb-df70-4c82-b805-f93bcc7e8030/network-check-target-container/0.log" Apr 16 14:22:46.096394 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:46.096363 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-v2ntl_7d2abdf5-4927-45c4-9386-af7dc0ed8d90/iptables-alerter/0.log" Apr 16 14:22:46.799479 ip-10-0-136-114 kubenswrapper[2574]: I0416 14:22:46.799450 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-6srr9_bbd94c79-236c-4a37-b6ad-384d38491b9e/tuned/0.log"