I0424 12:20:06.730209 1 start.go:62] Version: 7cb990c2 (6fd31b7e48fefb01d263cd19b733aca57ba2cf95) I0424 12:20:06.730689 1 leaderelection.go:121] The leader election gives 4 retries and allows for 30s of clock skew. The kube-apiserver downtime tolerance is 78s. Worst non-graceful lease acquisition is 2m43s. Worst graceful lease acquisition is {26s}. I0424 12:20:06.730851 1 envvar.go:172] "Feature gate default state" feature="ClientsAllowCBOR" enabled=false I0424 12:20:06.730878 1 envvar.go:172] "Feature gate default state" feature="ClientsPreferCBOR" enabled=false I0424 12:20:06.730891 1 envvar.go:172] "Feature gate default state" feature="InformerResourceVersion" enabled=false I0424 12:20:06.730910 1 envvar.go:172] "Feature gate default state" feature="WatchListClient" enabled=false I0424 12:20:06.863768 1 leaderelection.go:257] attempting to acquire leader lease openshift-machine-config-operator/machine-config-controller... I0424 12:22:28.746684 1 leaderelection.go:271] successfully acquired lease openshift-machine-config-operator/machine-config-controller I0424 12:22:28.759340 1 metrics.go:92] Registering Prometheus metrics I0424 12:22:28.759410 1 metrics.go:99] Starting metrics listener on 127.0.0.1:8797 I0424 12:22:28.759558 1 simple_featuregate_reader.go:171] Starting feature-gate-detector I0424 12:22:28.769020 1 certrotation_controller.go:170] Adding MCS CA/TLS cert rotator I0424 12:22:28.772720 1 reflector.go:376] Caches populated for *v1.ControllerConfig from github.com/openshift/client-go/machineconfiguration/informers/externalversions/factory.go:125 I0424 12:22:28.776094 1 reflector.go:376] Caches populated for *v1.Infrastructure from github.com/openshift/client-go/config/informers/externalversions/factory.go:125 I0424 12:22:28.776142 1 reflector.go:376] Caches populated for *v1.Image from github.com/openshift/client-go/config/informers/externalversions/factory.go:125 I0424 12:22:28.776552 1 reflector.go:376] Caches populated for *v1.ImageTagMirrorSet from github.com/openshift/client-go/config/informers/externalversions/factory.go:125 I0424 12:22:28.776586 1 reflector.go:376] Caches populated for *v1.Secret from k8s.io/client-go/informers/factory.go:160 I0424 12:22:28.776846 1 reflector.go:376] Caches populated for *v1alpha1.ImageContentSourcePolicy from github.com/openshift/client-go/operator/informers/externalversions/factory.go:125 I0424 12:22:28.776785 1 reflector.go:376] Caches populated for *v1.ImageDigestMirrorSet from github.com/openshift/client-go/config/informers/externalversions/factory.go:125 I0424 12:22:28.777749 1 reflector.go:376] Caches populated for *v1.APIServer from github.com/openshift/client-go/config/informers/externalversions/factory.go:125 I0424 12:22:28.777863 1 template_controller.go:198] Re-syncing ControllerConfig due to apiServer cluster change I0424 12:22:28.777882 1 kubelet_config_controller.go:222] Re-syncing all kubelet config controller generated MachineConfigs due to apiServer cluster change I0424 12:22:28.777937 1 reflector.go:376] Caches populated for *v1.MachineConfigPool from github.com/openshift/client-go/machineconfiguration/informers/externalversions/factory.go:125 I0424 12:22:28.778860 1 reflector.go:376] Caches populated for *v1.Node from k8s.io/client-go/informers/factory.go:160 I0424 12:22:28.780039 1 reflector.go:376] Caches populated for *v1.FeatureGate from github.com/openshift/client-go/config/informers/externalversions/factory.go:125 I0424 12:22:28.780222 1 reflector.go:376] Caches populated for *v1.ConfigMap from k8s.io/client-go/informers/factory.go:160 I0424 12:22:28.780331 1 reflector.go:376] Caches populated for *v1.Secret from k8s.io/client-go/informers/factory.go:160 I0424 12:22:28.781689 1 reflector.go:376] Caches populated for *v1.Secret from k8s.io/client-go/informers/factory.go:160 I0424 12:22:28.781822 1 start.go:121] FeatureGates initialized: enabled=[AWSEFSDriverVolumeMetrics AdditionalRoutingCapabilities AdminNetworkPolicy AlibabaPlatform AzureWorkloadIdentity BareMetalLoadBalancer BuildCSIVolumes CPMSMachineNamePrefix ChunkSizeMiB CloudDualStackNodeIPs ConsolePluginContentSecurityPolicy DisableKubeletCloudCredentialProviders GCPLabelsTags GatewayAPI GatewayAPIController HardwareSpeed IngressControllerLBSubnetsAWS KMSv1 MachineConfigNodes ManagedBootImages ManagedBootImagesAWS MetricsCollectionProfiles MultiArchInstallAWS MultiArchInstallGCP NetworkDiagnosticsConfig NetworkLiveMigration NetworkSegmentation NewOLM NodeDisruptionPolicy OnClusterBuild PersistentIPsForVirtualization PinnedImages PrivateHostedZoneAWS RouteAdvertisements RouteExternalCertificate ServiceAccountTokenNodeBinding SetEIPForNLBIngressController VSphereDriverConfiguration VSphereMultiVCenters ValidatingAdmissionPolicy] disabled=[AWSClusterHostedDNS AutomatedEtcdBackup BootcNodeManagement ClusterAPIInstall ClusterAPIInstallIBMCloud ClusterMonitoringConfig ClusterVersionOperatorConfiguration DNSNameResolver DualReplica DyanmicServiceEndpointIBMCloud DynamicResourceAllocation EtcdBackendQuota EventedPLEG Example Example2 ExternalOIDC ExternalOIDCWithUIDAndExtraClaimMappings GCPClusterHostedDNS GCPCustomAPIEndpoints HighlyAvailableArbiter ImageStreamImportMode IngressControllerDynamicConfigurationManager InsightsConfig InsightsConfigAPI InsightsOnDemandDataGather InsightsRuntimeExtractor KMSEncryptionProvider MachineAPIMigration MachineAPIOperatorDisableMachineHealthCheckController MachineAPIProviderOpenStack MaxUnavailableStatefulSet MinimumKubeletVersion MixedCPUsAllocation MultiArchInstallAzure NewOLMCatalogdAPIV1Metas NewOLMOwnSingleNamespace NewOLMPreflightPermissionChecks NodeSwap NutanixMultiSubnets OVNObservability OpenShiftPodSecurityAdmission PlatformOperators ProcMountType SELinuxChangePolicy SELinuxMount ShortCertRotation SignatureStores SigstoreImageVerification SigstoreImageVerificationPKI StreamingCollectionEncodingToJSON StreamingCollectionEncodingToProtobuf TranslateStreamCloseWebsocketRequests UpgradeStatus UserNamespacesPodSecurityStandards UserNamespacesSupport VSphereConfigurableMaxAllowedBlockVolumesPerNode VSphereHostVMGroupZonal VSphereMultiDisk VSphereMultiNetworks VolumeAttributesClass VolumeGroupSnapshot] I0424 12:22:28.782935 1 event.go:377] Event(v1.ObjectReference{Kind:"Node", Namespace:"openshift-machine-config-operator", Name:"ip-10-0-15-10.ec2.internal", UID:"e6c237b1-b8d8-4971-9651-8675c2ab8cba", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'FeatureGatesInitialized' FeatureGates updated to featuregates.Features{Enabled:[]v1.FeatureGateName{"AWSEFSDriverVolumeMetrics", "AdditionalRoutingCapabilities", "AdminNetworkPolicy", "AlibabaPlatform", "AzureWorkloadIdentity", "BareMetalLoadBalancer", "BuildCSIVolumes", "CPMSMachineNamePrefix", "ChunkSizeMiB", "CloudDualStackNodeIPs", "ConsolePluginContentSecurityPolicy", "DisableKubeletCloudCredentialProviders", "GCPLabelsTags", "GatewayAPI", "GatewayAPIController", "HardwareSpeed", "IngressControllerLBSubnetsAWS", "KMSv1", "MachineConfigNodes", "ManagedBootImages", "ManagedBootImagesAWS", "MetricsCollectionProfiles", "MultiArchInstallAWS", "MultiArchInstallGCP", "NetworkDiagnosticsConfig", "NetworkLiveMigration", "NetworkSegmentation", "NewOLM", "NodeDisruptionPolicy", "OnClusterBuild", "PersistentIPsForVirtualization", "PinnedImages", "PrivateHostedZoneAWS", "RouteAdvertisements", "RouteExternalCertificate", "ServiceAccountTokenNodeBinding", "SetEIPForNLBIngressController", "VSphereDriverConfiguration", "VSphereMultiVCenters", "ValidatingAdmissionPolicy"}, Disabled:[]v1.FeatureGateName{"AWSClusterHostedDNS", "AutomatedEtcdBackup", "BootcNodeManagement", "ClusterAPIInstall", "ClusterAPIInstallIBMCloud", "ClusterMonitoringConfig", "ClusterVersionOperatorConfiguration", "DNSNameResolver", "DualReplica", "DyanmicServiceEndpointIBMCloud", "DynamicResourceAllocation", "EtcdBackendQuota", "EventedPLEG", "Example", "Example2", "ExternalOIDC", "ExternalOIDCWithUIDAndExtraClaimMappings", "GCPClusterHostedDNS", "GCPCustomAPIEndpoints", "HighlyAvailableArbiter", "ImageStreamImportMode", "IngressControllerDynamicConfigurationManager", "InsightsConfig", "InsightsConfigAPI", "InsightsOnDemandDataGather", "InsightsRuntimeExtractor", "KMSEncryptionProvider", "MachineAPIMigration", "MachineAPIOperatorDisableMachineHealthCheckController", "MachineAPIProviderOpenStack", "MaxUnavailableStatefulSet", "MinimumKubeletVersion", "MixedCPUsAllocation", "MultiArchInstallAzure", "NewOLMCatalogdAPIV1Metas", "NewOLMOwnSingleNamespace", "NewOLMPreflightPermissionChecks", "NodeSwap", "NutanixMultiSubnets", "OVNObservability", "OpenShiftPodSecurityAdmission", "PlatformOperators", "ProcMountType", "SELinuxChangePolicy", "SELinuxMount", "ShortCertRotation", "SignatureStores", "SigstoreImageVerification", "SigstoreImageVerificationPKI", "StreamingCollectionEncodingToJSON", "StreamingCollectionEncodingToProtobuf", "TranslateStreamCloseWebsocketRequests", "UpgradeStatus", "UserNamespacesPodSecurityStandards", "UserNamespacesSupport", "VSphereConfigurableMaxAllowedBlockVolumesPerNode", "VSphereHostVMGroupZonal", "VSphereMultiDisk", "VSphereMultiNetworks", "VolumeAttributesClass", "VolumeGroupSnapshot"}} I0424 12:22:28.783087 1 template_controller.go:146] Re-syncing ControllerConfig due to secret pull-secret change E0424 12:22:28.783945 1 node_controller.go:536] getting scheduler config failed: cluster scheduler couldn't be found E0424 12:22:28.784051 1 node_controller.go:536] getting scheduler config failed: cluster scheduler couldn't be found E0424 12:22:28.784079 1 node_controller.go:536] getting scheduler config failed: cluster scheduler couldn't be found I0424 12:22:28.784366 1 reflector.go:376] Caches populated for *v1.ClusterVersion from github.com/openshift/client-go/config/informers/externalversions/factory.go:125 I0424 12:22:28.791310 1 reflector.go:376] Caches populated for *v1.PinnedImageSet from github.com/openshift/client-go/machineconfiguration/informers/externalversions/factory.go:125 I0424 12:22:28.794223 1 reflector.go:376] Caches populated for *v1.KubeletConfig from github.com/openshift/client-go/machineconfiguration/informers/externalversions/factory.go:125 I0424 12:22:28.801411 1 certrotation_controller.go:192] Starting machineconfigcontroller-certrotationcontroller I0424 12:22:28.801425 1 certrotation_controller.go:180] Waiting for machineconfigcontroller-certrotationcontroller I0424 12:22:28.801432 1 certrotation_controller.go:188] Finished waiting for machineconfigcontroller-certrotationcontroller I0424 12:22:28.801763 1 machine_set_boot_image_controller.go:233] configMap coreos-bootimages added, reconciling enrolled machine resources I0424 12:22:28.802208 1 drain_controller.go:179] Starting MachineConfigController-DrainController E0424 12:22:28.802272 1 machine_set_boot_image_controller.go:347] MachineConfiguration/cluster has not been created yet I0424 12:22:28.802318 1 kubelet_config_controller.go:201] Starting MachineConfigController-KubeletConfigController I0424 12:22:28.804222 1 reflector.go:376] Caches populated for *v1.ContainerRuntimeConfig from github.com/openshift/client-go/machineconfiguration/informers/externalversions/factory.go:125 I0424 12:22:28.805821 1 hostnames.go:43] syncing hostnames: [api-int.clusterpool419-52vmb.rhtap.devcluster.openshift.com] I0424 12:22:28.805885 1 base_controller.go:76] Waiting for caches to sync for CertRotationController I0424 12:22:28.805906 1 base_controller.go:82] Caches are synced for CertRotationController I0424 12:22:28.805923 1 base_controller.go:119] Starting #1 worker of CertRotationController controller ... I0424 12:22:28.805971 1 certrotation_controller.go:334] configMap machine-config-server-ca added, reconciling all user data secrets I0424 12:22:28.806045 1 certrotation_controller.go:373] secret worker-user-data added, ensuring CA is up to date I0424 12:22:28.806096 1 certrotation_controller.go:373] secret master-user-data added, ensuring CA is up to date I0424 12:22:28.806153 1 hostnames.go:43] syncing hostnames: [api-int.clusterpool419-52vmb.rhtap.devcluster.openshift.com] I0424 12:22:28.807469 1 reflector.go:376] Caches populated for *v1beta1.MachineSet from github.com/openshift/client-go/machine/informers/externalversions/factory.go:125 I0424 12:22:28.807622 1 machine_set_boot_image_controller.go:183] MAPI MachineSet clusterpool419-52vmb-8j62z-worker-us-east-1a added, reconciling enrolled machine resources I0424 12:22:28.807636 1 machine_set_boot_image_controller.go:183] MAPI MachineSet clusterpool419-52vmb-8j62z-worker-us-east-1b added, reconciling enrolled machine resources I0424 12:22:28.807645 1 machine_set_boot_image_controller.go:183] MAPI MachineSet clusterpool419-52vmb-8j62z-worker-us-east-1c added, reconciling enrolled machine resources I0424 12:22:28.807660 1 machine_set_boot_image_controller.go:183] MAPI MachineSet clusterpool419-52vmb-8j62z-worker-us-east-1d added, reconciling enrolled machine resources I0424 12:22:28.807667 1 machine_set_boot_image_controller.go:183] MAPI MachineSet clusterpool419-52vmb-8j62z-worker-us-east-1e added, reconciling enrolled machine resources I0424 12:22:28.807673 1 machine_set_boot_image_controller.go:183] MAPI MachineSet clusterpool419-52vmb-8j62z-worker-us-east-1f added, reconciling enrolled machine resources I0424 12:22:28.811593 1 reflector.go:376] Caches populated for *v1.MachineConfig from github.com/openshift/client-go/machineconfiguration/informers/externalversions/factory.go:125 I0424 12:22:28.819348 1 reflector.go:376] Caches populated for *v1.Node from github.com/openshift/client-go/config/informers/externalversions/factory.go:125 I0424 12:22:28.829883 1 reflector.go:376] Caches populated for *v1.Scheduler from github.com/openshift/client-go/config/informers/externalversions/factory.go:125 I0424 12:22:28.831140 1 reflector.go:376] Caches populated for *v1.Pod from k8s.io/client-go/informers/factory.go:160 I0424 12:22:28.868779 1 reflector.go:376] Caches populated for *v1.MachineConfiguration from github.com/openshift/client-go/operator/informers/externalversions/factory.go:125 I0424 12:22:28.868889 1 machine_set_boot_image_controller.go:287] Bootimages management configuration has been added, reconciling enrolled machine resources I0424 12:22:28.882735 1 reflector.go:376] Caches populated for *v1.MachineOSConfig from github.com/openshift/client-go/machineconfiguration/informers/externalversions/factory.go:125 I0424 12:22:28.883644 1 pinned_image_set.go:115] Starting MachineConfigController-PinnedImageSetController I0424 12:22:28.902457 1 node_controller.go:250] Starting MachineConfigController-NodeController I0424 12:22:28.902476 1 container_runtime_config_controller.go:247] Starting MachineConfigController-ContainerRuntimeConfigController I0424 12:22:28.902460 1 machine_set_boot_image_controller.go:173] Starting MachineConfigController-MachineSetBootImageController I0424 12:22:28.902492 1 template_controller.go:294] Starting MachineConfigController-TemplateController I0424 12:22:28.902480 1 render_controller.go:141] Starting MachineConfigController-RenderController I0424 12:22:29.303058 1 kubelet_config_nodes.go:163] Applied Node configuration 97-master-generated-kubelet on MachineConfigPool master I0424 12:22:29.312814 1 kubelet_config_features.go:126] Applied FeatureSet cluster on MachineConfigPool master I0424 12:22:29.609721 1 kubelet_config_nodes.go:163] Applied Node configuration 97-worker-generated-kubelet on MachineConfigPool worker I0424 12:22:29.810612 1 kubelet_config_features.go:126] Applied FeatureSet cluster on MachineConfigPool worker I0424 12:22:30.809644 1 kubelet_config_features.go:126] Applied FeatureSet cluster on MachineConfigPool master I0424 12:22:31.609318 1 kubelet_config_features.go:126] Applied FeatureSet cluster on MachineConfigPool worker I0424 12:22:33.409013 1 kubelet_config_nodes.go:163] Applied Node configuration 97-master-generated-kubelet on MachineConfigPool master I0424 12:22:33.830880 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1a on AWS, with arch x86_64 I0424 12:22:33.832433 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1a I0424 12:22:33.870189 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1b on AWS, with arch x86_64 I0424 12:22:33.871595 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1b I0424 12:22:33.908118 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1c on AWS, with arch x86_64 I0424 12:22:33.909368 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1c I0424 12:22:34.009305 1 kubelet_config_nodes.go:163] Applied Node configuration 97-worker-generated-kubelet on MachineConfigPool worker I0424 12:22:34.218005 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1d on AWS, with arch x86_64 I0424 12:22:34.218779 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1d I0424 12:22:34.609494 1 kubelet_config_features.go:126] Applied FeatureSet cluster on MachineConfigPool master I0424 12:22:34.817304 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1e on AWS, with arch x86_64 I0424 12:22:34.818510 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1e I0424 12:22:35.208878 1 kubelet_config_features.go:126] Applied FeatureSet cluster on MachineConfigPool worker I0424 12:22:35.416879 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1f on AWS, with arch x86_64 I0424 12:22:35.418089 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1f I0424 12:22:37.217992 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1f on AWS, with arch x86_64 I0424 12:22:37.219174 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1f I0424 12:22:37.816659 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1a on AWS, with arch x86_64 I0424 12:22:37.817796 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1a I0424 12:22:38.423050 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1b on AWS, with arch x86_64 I0424 12:22:38.424362 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1b I0424 12:22:39.021396 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1c on AWS, with arch x86_64 I0424 12:22:39.023497 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1c I0424 12:22:39.616666 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1d on AWS, with arch x86_64 I0424 12:22:39.617700 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1d I0424 12:22:40.216280 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1e on AWS, with arch x86_64 I0424 12:22:40.217185 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1e I0424 12:22:42.016999 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1b on AWS, with arch x86_64 I0424 12:22:42.017886 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1b I0424 12:22:42.617665 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1c on AWS, with arch x86_64 I0424 12:22:42.618526 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1c I0424 12:22:43.217547 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1d on AWS, with arch x86_64 I0424 12:22:43.218421 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1d I0424 12:22:43.821575 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1e on AWS, with arch x86_64 I0424 12:22:43.823526 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1e I0424 12:22:44.417146 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1f on AWS, with arch x86_64 I0424 12:22:44.418075 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1f I0424 12:22:45.017520 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1a on AWS, with arch x86_64 I0424 12:22:45.018265 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1a I0424 12:22:46.417295 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1a on AWS, with arch x86_64 I0424 12:22:46.418149 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1a I0424 12:22:47.016132 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1b on AWS, with arch x86_64 I0424 12:22:47.017049 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1b I0424 12:22:47.618105 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1c on AWS, with arch x86_64 I0424 12:22:47.618976 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1c I0424 12:22:48.217068 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1d on AWS, with arch x86_64 I0424 12:22:48.218270 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1d I0424 12:22:48.816307 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1e on AWS, with arch x86_64 I0424 12:22:48.817242 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1e I0424 12:22:49.419743 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1f on AWS, with arch x86_64 I0424 12:22:49.420431 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1f I0424 12:22:50.816246 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1c on AWS, with arch x86_64 I0424 12:22:50.817103 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1c I0424 12:22:51.417185 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1d on AWS, with arch x86_64 I0424 12:22:51.418205 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1d I0424 12:22:52.017239 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1e on AWS, with arch x86_64 I0424 12:22:52.018067 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1e I0424 12:22:52.615550 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1f on AWS, with arch x86_64 I0424 12:22:52.616288 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1f I0424 12:22:53.215864 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1a on AWS, with arch x86_64 I0424 12:22:53.216973 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1a I0424 12:22:53.816339 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1b on AWS, with arch x86_64 I0424 12:22:53.817330 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1b I0424 12:22:55.216828 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1c on AWS, with arch x86_64 I0424 12:22:55.217745 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1c I0424 12:22:55.816136 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1d on AWS, with arch x86_64 I0424 12:22:55.817035 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1d I0424 12:22:56.424360 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1e on AWS, with arch x86_64 I0424 12:22:56.425496 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1e I0424 12:22:57.016130 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1f on AWS, with arch x86_64 I0424 12:22:57.017229 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1f I0424 12:22:57.614451 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1a on AWS, with arch x86_64 I0424 12:22:57.615692 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1a I0424 12:22:58.214861 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1b on AWS, with arch x86_64 I0424 12:22:58.215810 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1b I0424 12:22:59.616391 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1c on AWS, with arch x86_64 I0424 12:22:59.617465 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1c I0424 12:23:00.218826 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1d on AWS, with arch x86_64 I0424 12:23:00.219917 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1d I0424 12:23:00.836556 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1e on AWS, with arch x86_64 I0424 12:23:00.837895 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1e I0424 12:23:01.431875 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1f on AWS, with arch x86_64 I0424 12:23:01.456167 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1f I0424 12:23:02.020541 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1a on AWS, with arch x86_64 I0424 12:23:02.021744 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1a I0424 12:23:02.622364 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1b on AWS, with arch x86_64 I0424 12:23:02.623853 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1b I0424 12:23:04.028112 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1b on AWS, with arch x86_64 I0424 12:23:04.029257 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1b I0424 12:23:04.645091 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1c on AWS, with arch x86_64 I0424 12:23:04.646356 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1c I0424 12:23:05.227957 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1d on AWS, with arch x86_64 I0424 12:23:05.229204 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1d I0424 12:23:05.836729 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1e on AWS, with arch x86_64 I0424 12:23:05.839530 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1e I0424 12:23:06.419600 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1f on AWS, with arch x86_64 I0424 12:23:06.420927 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1f I0424 12:23:07.017837 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1a on AWS, with arch x86_64 I0424 12:23:07.019658 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-52vmb-8j62z-worker-us-east-1a I0424 12:23:10.526472 1 render_controller.go:560] Generated machineconfig rendered-master-66fc81b3e21b61e94d4687244e2736a9 from 6 configs: [{MachineConfig 00-master machineconfiguration.openshift.io/v1 } {MachineConfig 01-master-container-runtime machineconfiguration.openshift.io/v1 } {MachineConfig 01-master-kubelet machineconfiguration.openshift.io/v1 } {MachineConfig 97-master-generated-kubelet machineconfiguration.openshift.io/v1 } {MachineConfig 98-master-generated-kubelet machineconfiguration.openshift.io/v1 } {MachineConfig 99-master-generated-registries machineconfiguration.openshift.io/v1 }] I0424 12:23:10.527103 1 event.go:377] Event(v1.ObjectReference{Kind:"MachineConfigPool", Namespace:"openshift-machine-config-operator", Name:"master", UID:"0489bf1a-c9e3-422d-aa7d-18e7a20e709d", APIVersion:"machineconfiguration.openshift.io/v1", ResourceVersion:"41258", FieldPath:""}): type: 'Normal' reason: 'RenderedConfigGenerated' rendered-master-66fc81b3e21b61e94d4687244e2736a9 successfully generated (release version: 4.19.21, controller version: 6fd31b7e48fefb01d263cd19b733aca57ba2cf95) I0424 12:23:10.541924 1 render_controller.go:586] Pool master: now targeting: rendered-master-66fc81b3e21b61e94d4687244e2736a9 I0424 12:23:10.593292 1 render_controller.go:560] Generated machineconfig rendered-worker-ac034372faedd2580c791a8efdf5a593 from 6 configs: [{MachineConfig 00-worker machineconfiguration.openshift.io/v1 } {MachineConfig 01-worker-container-runtime machineconfiguration.openshift.io/v1 } {MachineConfig 01-worker-kubelet machineconfiguration.openshift.io/v1 } {MachineConfig 97-worker-generated-kubelet machineconfiguration.openshift.io/v1 } {MachineConfig 98-worker-generated-kubelet machineconfiguration.openshift.io/v1 } {MachineConfig 99-worker-generated-registries machineconfiguration.openshift.io/v1 }] I0424 12:23:10.593938 1 event.go:377] Event(v1.ObjectReference{Kind:"MachineConfigPool", Namespace:"openshift-machine-config-operator", Name:"worker", UID:"82dda715-459c-4c69-87f8-bb6e76d8ccd1", APIVersion:"machineconfiguration.openshift.io/v1", ResourceVersion:"26097", FieldPath:""}): type: 'Normal' reason: 'RenderedConfigGenerated' rendered-worker-ac034372faedd2580c791a8efdf5a593 successfully generated (release version: 4.19.21, controller version: 6fd31b7e48fefb01d263cd19b733aca57ba2cf95) I0424 12:23:10.608397 1 render_controller.go:586] Pool worker: now targeting: rendered-worker-ac034372faedd2580c791a8efdf5a593 I0424 12:23:15.574292 1 node_controller.go:607] Pool master[zone=us-east-1a]: node ip-10-0-15-10.ec2.internal: changed taints I0424 12:23:15.618721 1 node_controller.go:607] Pool master[zone=us-east-1b]: node ip-10-0-60-165.ec2.internal: changed taints I0424 12:23:15.676332 1 node_controller.go:1308] Pool master: selected candidate node ip-10-0-15-10.ec2.internal I0424 12:23:15.676409 1 node_controller.go:1308] Pool master: selected candidate node ip-10-0-60-165.ec2.internal I0424 12:23:15.676426 1 node_controller.go:1308] Pool master: selected candidate node ip-10-0-94-193.ec2.internal I0424 12:23:15.676486 1 node_controller.go:597] Pool master: 3 candidate nodes in 3 zones for update, capacity: 1 I0424 12:23:15.676635 1 node_controller.go:1357] Deferring update of machine config operator node: ip-10-0-94-193.ec2.internal I0424 12:23:15.676654 1 node_controller.go:597] Pool master: filtered to 2 candidate nodes for update, capacity: 1 I0424 12:23:15.676913 1 event.go:377] Event(v1.ObjectReference{Kind:"MachineConfigPool", Namespace:"openshift-machine-config-operator", Name:"master", UID:"0489bf1a-c9e3-422d-aa7d-18e7a20e709d", APIVersion:"machineconfiguration.openshift.io/v1", ResourceVersion:"42208", FieldPath:""}): type: 'Normal' reason: 'DeferringOperatorNodeUpdate' Deferring update of machine config operator node ip-10-0-94-193.ec2.internal I0424 12:23:15.677167 1 node_controller.go:607] Pool master[zone=us-east-1c]: node ip-10-0-94-193.ec2.internal: changed taints I0424 12:23:15.709260 1 node_controller.go:607] Pool worker[zone=us-east-1a]: node ip-10-0-22-16.ec2.internal: changed taints I0424 12:23:15.965929 1 node_controller.go:607] Pool master[zone=us-east-1a]: node ip-10-0-15-10.ec2.internal: changed annotation machineconfiguration.openshift.io/desiredConfig = rendered-master-66fc81b3e21b61e94d4687244e2736a9 I0424 12:23:15.966110 1 event.go:377] Event(v1.ObjectReference{Kind:"MachineConfigPool", Namespace:"openshift-machine-config-operator", Name:"master", UID:"0489bf1a-c9e3-422d-aa7d-18e7a20e709d", APIVersion:"machineconfiguration.openshift.io/v1", ResourceVersion:"42208", FieldPath:""}): type: 'Normal' reason: 'SetDesiredConfig' Targeted node ip-10-0-15-10.ec2.internal to MachineConfig: rendered-master-66fc81b3e21b61e94d4687244e2736a9 I0424 12:23:15.966151 1 event.go:377] Event(v1.ObjectReference{Kind:"MachineConfigPool", Namespace:"openshift-machine-config-operator", Name:"master", UID:"0489bf1a-c9e3-422d-aa7d-18e7a20e709d", APIVersion:"machineconfiguration.openshift.io/v1", ResourceVersion:"42208", FieldPath:""}): type: 'Normal' reason: 'AnnotationChange' Node ip-10-0-15-10.ec2.internal now has machineconfiguration.openshift.io/desiredConfig=rendered-master-66fc81b3e21b61e94d4687244e2736a9 I0424 12:23:16.162534 1 node_controller.go:607] Pool worker[zone=us-east-1b]: node ip-10-0-34-100.ec2.internal: changed taints I0424 12:23:16.882198 1 node_controller.go:607] Pool master[zone=us-east-1a]: node ip-10-0-15-10.ec2.internal: changed annotation machineconfiguration.openshift.io/state = Working I0424 12:23:16.882372 1 event.go:377] Event(v1.ObjectReference{Kind:"MachineConfigPool", Namespace:"openshift-machine-config-operator", Name:"master", UID:"0489bf1a-c9e3-422d-aa7d-18e7a20e709d", APIVersion:"machineconfiguration.openshift.io/v1", ResourceVersion:"42439", FieldPath:""}): type: 'Normal' reason: 'AnnotationChange' Node ip-10-0-15-10.ec2.internal now has machineconfiguration.openshift.io/state=Working I0424 12:23:16.967364 1 node_controller.go:607] Pool worker[zone=us-east-1c]: node ip-10-0-71-88.ec2.internal: changed taints I0424 12:23:16.972571 1 node_controller.go:1308] Pool worker: selected candidate node ip-10-0-22-16.ec2.internal I0424 12:23:16.972587 1 node_controller.go:1308] Pool worker: selected candidate node ip-10-0-34-100.ec2.internal I0424 12:23:16.972592 1 node_controller.go:1308] Pool worker: selected candidate node ip-10-0-71-88.ec2.internal I0424 12:23:16.972602 1 node_controller.go:597] Pool worker: 3 candidate nodes in 3 zones for update, capacity: 1 I0424 12:23:17.561990 1 node_controller.go:607] Pool worker[zone=us-east-1a]: node ip-10-0-22-16.ec2.internal: changed annotation machineconfiguration.openshift.io/desiredConfig = rendered-worker-ac034372faedd2580c791a8efdf5a593 I0424 12:23:17.562107 1 event.go:377] Event(v1.ObjectReference{Kind:"MachineConfigPool", Namespace:"openshift-machine-config-operator", Name:"worker", UID:"82dda715-459c-4c69-87f8-bb6e76d8ccd1", APIVersion:"machineconfiguration.openshift.io/v1", ResourceVersion:"42214", FieldPath:""}): type: 'Normal' reason: 'SetDesiredConfig' Targeted node ip-10-0-22-16.ec2.internal to MachineConfig: rendered-worker-ac034372faedd2580c791a8efdf5a593 I0424 12:23:18.756312 1 node_controller.go:607] Pool worker[zone=us-east-1a]: node ip-10-0-22-16.ec2.internal: changed annotation machineconfiguration.openshift.io/state = Working I0424 12:23:20.611012 1 node_controller.go:607] Pool master[zone=us-east-1a]: node ip-10-0-15-10.ec2.internal: changed taints I0424 12:23:20.807864 1 node_controller.go:607] Pool worker[zone=us-east-1a]: node ip-10-0-22-16.ec2.internal: changed taints I0424 12:23:32.392349 1 node_controller.go:607] Pool worker[zone=us-east-1a]: node ip-10-0-22-16.ec2.internal: Reporting unready: node ip-10-0-22-16.ec2.internal is reporting NotReady=False I0424 12:23:32.437287 1 node_controller.go:607] Pool worker[zone=us-east-1a]: node ip-10-0-22-16.ec2.internal: changed taints I0424 12:23:37.080522 1 drain_controller.go:193] node ip-10-0-22-16.ec2.internal: uncordoning I0424 12:23:37.080583 1 drain_controller.go:193] node ip-10-0-22-16.ec2.internal: initiating uncordon (currently schedulable: true) I0424 12:23:37.086662 1 drain_controller.go:193] node ip-10-0-22-16.ec2.internal: uncordon succeeded (currently schedulable: true) I0424 12:23:37.113959 1 drain_controller.go:193] node ip-10-0-22-16.ec2.internal: operation successful; applying completion annotation I0424 12:23:37.470377 1 node_controller.go:607] Pool worker[zone=us-east-1a]: node ip-10-0-22-16.ec2.internal: changed taints I0424 12:23:42.133789 1 node_controller.go:607] Pool worker[zone=us-east-1a]: node ip-10-0-22-16.ec2.internal: Completed update to rendered-worker-ac034372faedd2580c791a8efdf5a593 I0424 12:23:42.428484 1 node_controller.go:607] Pool worker[zone=us-east-1a]: node ip-10-0-22-16.ec2.internal: Reporting ready I0424 12:23:42.448973 1 node_controller.go:607] Pool worker[zone=us-east-1a]: node ip-10-0-22-16.ec2.internal: changed taints I0424 12:23:42.488516 1 node_controller.go:1308] Pool worker: selected candidate node ip-10-0-34-100.ec2.internal I0424 12:23:42.488610 1 node_controller.go:1308] Pool worker: selected candidate node ip-10-0-71-88.ec2.internal I0424 12:23:42.488621 1 node_controller.go:597] Pool worker: 2 candidate nodes in 2 zones for update, capacity: 1 I0424 12:23:42.536487 1 node_controller.go:607] Pool worker[zone=us-east-1b]: node ip-10-0-34-100.ec2.internal: changed annotation machineconfiguration.openshift.io/desiredConfig = rendered-worker-ac034372faedd2580c791a8efdf5a593 I0424 12:23:42.536520 1 event.go:377] Event(v1.ObjectReference{Kind:"MachineConfigPool", Namespace:"openshift-machine-config-operator", Name:"worker", UID:"82dda715-459c-4c69-87f8-bb6e76d8ccd1", APIVersion:"machineconfiguration.openshift.io/v1", ResourceVersion:"42472", FieldPath:""}): type: 'Normal' reason: 'SetDesiredConfig' Targeted node ip-10-0-34-100.ec2.internal to MachineConfig: rendered-worker-ac034372faedd2580c791a8efdf5a593 I0424 12:23:43.708007 1 node_controller.go:607] Pool worker[zone=us-east-1b]: node ip-10-0-34-100.ec2.internal: changed annotation machineconfiguration.openshift.io/state = Working I0424 12:23:47.438348 1 node_controller.go:607] Pool worker[zone=us-east-1a]: node ip-10-0-22-16.ec2.internal: changed taints I0424 12:23:47.725636 1 node_controller.go:607] Pool worker[zone=us-east-1b]: node ip-10-0-34-100.ec2.internal: changed taints I0424 12:23:56.273159 1 node_controller.go:607] Pool worker[zone=us-east-1b]: node ip-10-0-34-100.ec2.internal: Reporting unready: node ip-10-0-34-100.ec2.internal is reporting NotReady=False I0424 12:23:56.303884 1 node_controller.go:607] Pool worker[zone=us-east-1b]: node ip-10-0-34-100.ec2.internal: changed taints I0424 12:23:57.489597 1 node_controller.go:607] Pool worker[zone=us-east-1b]: node ip-10-0-34-100.ec2.internal: changed taints I0424 12:24:00.894960 1 drain_controller.go:193] node ip-10-0-34-100.ec2.internal: uncordoning I0424 12:24:00.895012 1 drain_controller.go:193] node ip-10-0-34-100.ec2.internal: initiating uncordon (currently schedulable: true) I0424 12:24:00.910357 1 drain_controller.go:193] node ip-10-0-34-100.ec2.internal: uncordon succeeded (currently schedulable: true) I0424 12:24:00.965425 1 drain_controller.go:193] node ip-10-0-34-100.ec2.internal: operation successful; applying completion annotation I0424 12:24:05.978242 1 node_controller.go:607] Pool worker[zone=us-east-1b]: node ip-10-0-34-100.ec2.internal: Completed update to rendered-worker-ac034372faedd2580c791a8efdf5a593 I0424 12:24:06.519129 1 node_controller.go:607] Pool worker[zone=us-east-1b]: node ip-10-0-34-100.ec2.internal: Reporting ready I0424 12:24:06.548670 1 node_controller.go:607] Pool worker[zone=us-east-1b]: node ip-10-0-34-100.ec2.internal: changed taints I0424 12:24:07.463227 1 node_controller.go:607] Pool worker[zone=us-east-1b]: node ip-10-0-34-100.ec2.internal: changed taints I0424 12:24:11.006336 1 node_controller.go:1308] Pool worker: selected candidate node ip-10-0-71-88.ec2.internal I0424 12:24:11.006370 1 node_controller.go:597] Pool worker: 1 candidate nodes in 1 zones for update, capacity: 1 I0424 12:24:11.087388 1 event.go:377] Event(v1.ObjectReference{Kind:"MachineConfigPool", Namespace:"openshift-machine-config-operator", Name:"worker", UID:"82dda715-459c-4c69-87f8-bb6e76d8ccd1", APIVersion:"machineconfiguration.openshift.io/v1", ResourceVersion:"43864", FieldPath:""}): type: 'Normal' reason: 'SetDesiredConfig' Targeted node ip-10-0-71-88.ec2.internal to MachineConfig: rendered-worker-ac034372faedd2580c791a8efdf5a593 I0424 12:24:11.094636 1 node_controller.go:607] Pool worker[zone=us-east-1c]: node ip-10-0-71-88.ec2.internal: changed annotation machineconfiguration.openshift.io/desiredConfig = rendered-worker-ac034372faedd2580c791a8efdf5a593 I0424 12:24:12.300501 1 node_controller.go:607] Pool worker[zone=us-east-1c]: node ip-10-0-71-88.ec2.internal: changed annotation machineconfiguration.openshift.io/state = Working I0424 12:24:16.136579 1 node_controller.go:607] Pool worker[zone=us-east-1c]: node ip-10-0-71-88.ec2.internal: changed taints E0424 12:24:16.215884 1 pinned_image_set.go:350] Error syncing pinned image sets: Operation cannot be fulfilled on machineconfigpools.machineconfiguration.openshift.io "worker": the object has been modified; please apply your changes to the latest version and try again E0424 12:24:16.224062 1 pinned_image_set.go:373] Error updating MachineConfigPool worker: Operation cannot be fulfilled on machineconfigpools.machineconfiguration.openshift.io "worker": the object has been modified; please apply your changes to the latest version and try again I0424 12:24:16.224085 1 pinned_image_set.go:299] Error syncing machineconfigpool worker: Operation cannot be fulfilled on machineconfigpools.machineconfiguration.openshift.io "worker": the object has been modified; please apply your changes to the latest version and try again E0424 12:24:16.381044 1 render_controller.go:464] Error syncing Generated MCFG: Operation cannot be fulfilled on machineconfigpools.machineconfiguration.openshift.io "worker": the object has been modified; please apply your changes to the latest version and try again E0424 12:24:16.396274 1 render_controller.go:486] Error updating MachineConfigPool worker: Operation cannot be fulfilled on machineconfigpools.machineconfiguration.openshift.io "worker": the object has been modified; please apply your changes to the latest version and try again I0424 12:24:16.396297 1 render_controller.go:394] Error syncing machineconfigpool worker: Operation cannot be fulfilled on machineconfigpools.machineconfiguration.openshift.io "worker": the object has been modified; please apply your changes to the latest version and try again I0424 12:24:27.486747 1 node_controller.go:607] Pool master[zone=us-east-1a]: node ip-10-0-15-10.ec2.internal: Reporting unready: node ip-10-0-15-10.ec2.internal is reporting OutOfDisk=Unknown I0424 12:24:27.542713 1 node_controller.go:607] Pool master[zone=us-east-1a]: node ip-10-0-15-10.ec2.internal: changed taints I0424 12:24:27.933547 1 drain_controller.go:193] node ip-10-0-71-88.ec2.internal: uncordoning I0424 12:24:27.934255 1 drain_controller.go:193] node ip-10-0-71-88.ec2.internal: initiating uncordon (currently schedulable: true) I0424 12:24:27.955968 1 drain_controller.go:193] node ip-10-0-71-88.ec2.internal: uncordon succeeded (currently schedulable: true) I0424 12:24:28.003520 1 drain_controller.go:193] node ip-10-0-71-88.ec2.internal: operation successful; applying completion annotation I0424 12:24:32.982095 1 node_controller.go:607] Pool worker[zone=us-east-1c]: node ip-10-0-71-88.ec2.internal: Completed update to rendered-worker-ac034372faedd2580c791a8efdf5a593 I0424 12:24:35.195781 1 node_controller.go:607] Pool master[zone=us-east-1a]: node ip-10-0-15-10.ec2.internal: changed taints I0424 12:24:38.088886 1 status.go:266] Pool worker: All nodes are updated with MachineConfig rendered-worker-ac034372faedd2580c791a8efdf5a593 I0424 12:25:11.012738 1 node_controller.go:607] Pool master[zone=us-east-1a]: node ip-10-0-15-10.ec2.internal: Reporting ready I0424 12:25:11.310230 1 node_controller.go:607] Pool master[zone=us-east-1a]: node ip-10-0-15-10.ec2.internal: changed taints I0424 12:25:15.116432 1 node_controller.go:607] Pool master[zone=us-east-1a]: node ip-10-0-15-10.ec2.internal: changed taints I0424 12:25:15.542400 1 drain_controller.go:193] node ip-10-0-15-10.ec2.internal: uncordoning I0424 12:25:15.542518 1 drain_controller.go:193] node ip-10-0-15-10.ec2.internal: initiating uncordon (currently schedulable: true) I0424 12:25:15.547756 1 drain_controller.go:193] node ip-10-0-15-10.ec2.internal: uncordon succeeded (currently schedulable: true) I0424 12:25:15.565617 1 drain_controller.go:193] node ip-10-0-15-10.ec2.internal: operation successful; applying completion annotation I0424 12:25:20.563543 1 node_controller.go:607] Pool master[zone=us-east-1a]: node ip-10-0-15-10.ec2.internal: Completed update to rendered-master-66fc81b3e21b61e94d4687244e2736a9 I0424 12:25:25.569107 1 node_controller.go:1308] Pool master: selected candidate node ip-10-0-94-193.ec2.internal I0424 12:25:25.569157 1 node_controller.go:1308] Pool master: selected candidate node ip-10-0-60-165.ec2.internal I0424 12:25:25.569166 1 node_controller.go:597] Pool master: 2 candidate nodes in 2 zones for update, capacity: 1 I0424 12:25:25.569307 1 node_controller.go:1357] Deferring update of machine config operator node: ip-10-0-94-193.ec2.internal I0424 12:25:25.569318 1 node_controller.go:597] Pool master: filtered to 1 candidate nodes for update, capacity: 1 I0424 12:25:25.569391 1 event.go:377] Event(v1.ObjectReference{Kind:"MachineConfigPool", Namespace:"openshift-machine-config-operator", Name:"master", UID:"0489bf1a-c9e3-422d-aa7d-18e7a20e709d", APIVersion:"machineconfiguration.openshift.io/v1", ResourceVersion:"42439", FieldPath:""}): type: 'Normal' reason: 'DeferringOperatorNodeUpdate' Deferring update of machine config operator node ip-10-0-94-193.ec2.internal I0424 12:25:25.609292 1 event.go:377] Event(v1.ObjectReference{Kind:"MachineConfigPool", Namespace:"openshift-machine-config-operator", Name:"master", UID:"0489bf1a-c9e3-422d-aa7d-18e7a20e709d", APIVersion:"machineconfiguration.openshift.io/v1", ResourceVersion:"42439", FieldPath:""}): type: 'Normal' reason: 'SetDesiredConfig' Targeted node ip-10-0-60-165.ec2.internal to MachineConfig: rendered-master-66fc81b3e21b61e94d4687244e2736a9 I0424 12:25:25.612604 1 node_controller.go:607] Pool master[zone=us-east-1b]: node ip-10-0-60-165.ec2.internal: changed annotation machineconfiguration.openshift.io/desiredConfig = rendered-master-66fc81b3e21b61e94d4687244e2736a9 I0424 12:25:25.612751 1 event.go:377] Event(v1.ObjectReference{Kind:"MachineConfigPool", Namespace:"openshift-machine-config-operator", Name:"master", UID:"0489bf1a-c9e3-422d-aa7d-18e7a20e709d", APIVersion:"machineconfiguration.openshift.io/v1", ResourceVersion:"42439", FieldPath:""}): type: 'Normal' reason: 'AnnotationChange' Node ip-10-0-60-165.ec2.internal now has machineconfiguration.openshift.io/desiredConfig=rendered-master-66fc81b3e21b61e94d4687244e2736a9 I0424 12:25:26.825346 1 node_controller.go:607] Pool master[zone=us-east-1b]: node ip-10-0-60-165.ec2.internal: changed annotation machineconfiguration.openshift.io/state = Working I0424 12:25:26.825753 1 event.go:377] Event(v1.ObjectReference{Kind:"MachineConfigPool", Namespace:"openshift-machine-config-operator", Name:"master", UID:"0489bf1a-c9e3-422d-aa7d-18e7a20e709d", APIVersion:"machineconfiguration.openshift.io/v1", ResourceVersion:"50629", FieldPath:""}): type: 'Normal' reason: 'AnnotationChange' Node ip-10-0-60-165.ec2.internal now has machineconfiguration.openshift.io/state=Working I0424 12:25:30.656998 1 node_controller.go:607] Pool master[zone=us-east-1b]: node ip-10-0-60-165.ec2.internal: changed taints E0424 12:25:30.827840 1 render_controller.go:464] Error syncing Generated MCFG: Operation cannot be fulfilled on machineconfigpools.machineconfiguration.openshift.io "master": the object has been modified; please apply your changes to the latest version and try again E0424 12:25:30.834125 1 render_controller.go:486] Error updating MachineConfigPool master: Operation cannot be fulfilled on machineconfigpools.machineconfiguration.openshift.io "master": the object has been modified; please apply your changes to the latest version and try again I0424 12:25:30.834144 1 render_controller.go:394] Error syncing machineconfigpool master: Operation cannot be fulfilled on machineconfigpools.machineconfiguration.openshift.io "master": the object has been modified; please apply your changes to the latest version and try again I0424 12:26:30.142882 1 node_controller.go:607] Pool master[zone=us-east-1b]: node ip-10-0-60-165.ec2.internal: Reporting unready: node ip-10-0-60-165.ec2.internal is reporting OutOfDisk=Unknown I0424 12:26:30.172224 1 node_controller.go:607] Pool master[zone=us-east-1b]: node ip-10-0-60-165.ec2.internal: changed taints I0424 12:26:37.018145 1 node_controller.go:607] Pool master[zone=us-east-1b]: node ip-10-0-60-165.ec2.internal: changed taints I0424 12:27:12.741940 1 node_controller.go:607] Pool master[zone=us-east-1b]: node ip-10-0-60-165.ec2.internal: Reporting ready I0424 12:27:12.783319 1 node_controller.go:607] Pool master[zone=us-east-1b]: node ip-10-0-60-165.ec2.internal: changed taints I0424 12:27:17.017033 1 node_controller.go:607] Pool master[zone=us-east-1b]: node ip-10-0-60-165.ec2.internal: changed taints I0424 12:27:17.353048 1 drain_controller.go:193] node ip-10-0-60-165.ec2.internal: uncordoning I0424 12:27:17.353186 1 drain_controller.go:193] node ip-10-0-60-165.ec2.internal: initiating uncordon (currently schedulable: true) I0424 12:27:17.360724 1 drain_controller.go:193] node ip-10-0-60-165.ec2.internal: uncordon succeeded (currently schedulable: true) I0424 12:27:17.383958 1 drain_controller.go:193] node ip-10-0-60-165.ec2.internal: operation successful; applying completion annotation I0424 12:27:22.377879 1 node_controller.go:607] Pool master[zone=us-east-1b]: node ip-10-0-60-165.ec2.internal: Completed update to rendered-master-66fc81b3e21b61e94d4687244e2736a9 I0424 12:27:27.383776 1 node_controller.go:1308] Pool master: selected candidate node ip-10-0-94-193.ec2.internal I0424 12:27:27.383794 1 node_controller.go:597] Pool master: 1 candidate nodes in 1 zones for update, capacity: 1 I0424 12:27:27.383798 1 node_controller.go:597] Pool master: filtered to 1 candidate nodes for update, capacity: 1 I0424 12:27:27.418069 1 event.go:377] Event(v1.ObjectReference{Kind:"MachineConfigPool", Namespace:"openshift-machine-config-operator", Name:"master", UID:"0489bf1a-c9e3-422d-aa7d-18e7a20e709d", APIVersion:"machineconfiguration.openshift.io/v1", ResourceVersion:"50737", FieldPath:""}): type: 'Normal' reason: 'SetDesiredConfig' Targeted node ip-10-0-94-193.ec2.internal to MachineConfig: rendered-master-66fc81b3e21b61e94d4687244e2736a9 I0424 12:27:27.419095 1 node_controller.go:607] Pool master[zone=us-east-1c]: node ip-10-0-94-193.ec2.internal: changed annotation machineconfiguration.openshift.io/desiredConfig = rendered-master-66fc81b3e21b61e94d4687244e2736a9 I0424 12:27:27.419161 1 event.go:377] Event(v1.ObjectReference{Kind:"MachineConfigPool", Namespace:"openshift-machine-config-operator", Name:"master", UID:"0489bf1a-c9e3-422d-aa7d-18e7a20e709d", APIVersion:"machineconfiguration.openshift.io/v1", ResourceVersion:"50737", FieldPath:""}): type: 'Normal' reason: 'AnnotationChange' Node ip-10-0-94-193.ec2.internal now has machineconfiguration.openshift.io/desiredConfig=rendered-master-66fc81b3e21b61e94d4687244e2736a9 I0424 12:27:28.647203 1 node_controller.go:607] Pool master[zone=us-east-1c]: node ip-10-0-94-193.ec2.internal: changed annotation machineconfiguration.openshift.io/state = Working I0424 12:27:28.647550 1 event.go:377] Event(v1.ObjectReference{Kind:"MachineConfigPool", Namespace:"openshift-machine-config-operator", Name:"master", UID:"0489bf1a-c9e3-422d-aa7d-18e7a20e709d", APIVersion:"machineconfiguration.openshift.io/v1", ResourceVersion:"52700", FieldPath:""}): type: 'Normal' reason: 'AnnotationChange' Node ip-10-0-94-193.ec2.internal now has machineconfiguration.openshift.io/state=Working I0424 12:27:32.443242 1 node_controller.go:607] Pool master[zone=us-east-1c]: node ip-10-0-94-193.ec2.internal: changed taints E0424 12:27:32.627452 1 render_controller.go:464] Error syncing Generated MCFG: Operation cannot be fulfilled on machineconfigpools.machineconfiguration.openshift.io "master": the object has been modified; please apply your changes to the latest version and try again E0424 12:27:32.634695 1 render_controller.go:486] Error updating MachineConfigPool master: Operation cannot be fulfilled on machineconfigpools.machineconfiguration.openshift.io "master": the object has been modified; please apply your changes to the latest version and try again I0424 12:27:32.634719 1 render_controller.go:394] Error syncing machineconfigpool master: Operation cannot be fulfilled on machineconfigpools.machineconfiguration.openshift.io "master": the object has been modified; please apply your changes to the latest version and try again I0424 12:28:32.048206 1 node_controller.go:607] Pool master[zone=us-east-1c]: node ip-10-0-94-193.ec2.internal: Reporting unready: node ip-10-0-94-193.ec2.internal is reporting OutOfDisk=Unknown I0424 12:28:32.106083 1 node_controller.go:607] Pool master[zone=us-east-1c]: node ip-10-0-94-193.ec2.internal: changed taints I0424 12:28:40.045911 1 node_controller.go:607] Pool master[zone=us-east-1c]: node ip-10-0-94-193.ec2.internal: changed taints I0424 12:29:18.654593 1 node_controller.go:607] Pool master[zone=us-east-1c]: node ip-10-0-94-193.ec2.internal: Reporting ready I0424 12:29:18.682880 1 node_controller.go:607] Pool master[zone=us-east-1c]: node ip-10-0-94-193.ec2.internal: changed taints I0424 12:29:20.019689 1 node_controller.go:607] Pool master[zone=us-east-1c]: node ip-10-0-94-193.ec2.internal: changed taints I0424 12:29:23.118020 1 drain_controller.go:193] node ip-10-0-94-193.ec2.internal: uncordoning I0424 12:29:23.118078 1 drain_controller.go:193] node ip-10-0-94-193.ec2.internal: initiating uncordon (currently schedulable: true) I0424 12:29:23.123343 1 drain_controller.go:193] node ip-10-0-94-193.ec2.internal: uncordon succeeded (currently schedulable: true) I0424 12:29:23.141222 1 drain_controller.go:193] node ip-10-0-94-193.ec2.internal: operation successful; applying completion annotation I0424 12:29:28.159280 1 node_controller.go:607] Pool master[zone=us-east-1c]: node ip-10-0-94-193.ec2.internal: Completed update to rendered-master-66fc81b3e21b61e94d4687244e2736a9 I0424 12:29:33.184245 1 status.go:266] Pool master: All nodes are updated with MachineConfig rendered-master-66fc81b3e21b61e94d4687244e2736a9