I0428 12:54:18.269135       1 start.go:61] Version: 71361d19 (9e588a1318823dac89a6bc1790ca584d1100dd57)
I0428 12:54:18.269291       1 leaderelection.go:121] The leader election gives 4 retries and allows for 30s of clock skew. The kube-apiserver downtime tolerance is 78s. Worst non-graceful lease acquisition is 2m43s. Worst graceful lease acquisition is {26s}.
I0428 12:54:24.205400       1 leaderelection.go:254] attempting to acquire leader lease openshift-machine-config-operator/machine-config-controller...
I0428 12:54:26.131027       1 leaderelection.go:268] successfully acquired lease openshift-machine-config-operator/machine-config-controller
I0428 12:54:26.132166       1 envvar.go:172] "Feature gate default state" feature="WatchListClient" enabled=false
I0428 12:54:26.132228       1 envvar.go:172] "Feature gate default state" feature="InformerResourceVersion" enabled=false
I0428 12:54:26.227239       1 simple_featuregate_reader.go:171] Starting feature-gate-detector
I0428 12:54:26.227428       1 metrics.go:92] Registering Prometheus metrics
I0428 12:54:26.227536       1 metrics.go:99] Starting metrics listener on 127.0.0.1:8797
I0428 12:54:26.257192       1 reflector.go:368] Caches populated for *v1.ImageDigestMirrorSet from github.com/openshift/client-go/config/informers/externalversions/factory.go:125
I0428 12:54:26.261117       1 reflector.go:368] Caches populated for *v1.FeatureGate from github.com/openshift/client-go/config/informers/externalversions/factory.go:125
I0428 12:54:26.261959       1 start.go:105] FeatureGates initialized: enabled=[AWSEFSDriverVolumeMetrics AdminNetworkPolicy AlibabaPlatform AzureWorkloadIdentity BareMetalLoadBalancer BuildCSIVolumes ChunkSizeMiB CloudDualStackNodeIPs DisableKubeletCloudCredentialProviders GCPLabelsTags HardwareSpeed IngressControllerLBSubnetsAWS KMSv1 ManagedBootImages ManagedBootImagesAWS MultiArchInstallAWS MultiArchInstallGCP NetworkDiagnosticsConfig NetworkLiveMigration NetworkSegmentation NewOLM NodeDisruptionPolicy OnClusterBuild PersistentIPsForVirtualization PrivateHostedZoneAWS SetEIPForNLBIngressController StreamingCollectionEncodingToJSON StreamingCollectionEncodingToProtobuf VSphereControlPlaneMachineSet VSphereDriverConfiguration VSphereMultiVCenters VSphereStaticIPs ValidatingAdmissionPolicy]  disabled=[AWSClusterHostedDNS AdditionalRoutingCapabilities AutomatedEtcdBackup BootcNodeManagement CSIDriverSharedResource ClusterAPIInstall ClusterAPIInstallIBMCloud ClusterMonitoringConfig ConsolePluginContentSecurityPolicy DNSNameResolver DynamicResourceAllocation EtcdBackendQuota EventedPLEG Example ExternalOIDC ExternalOIDCWithUIDAndExtraClaimMappings GCPClusterHostedDNS GatewayAPI ImageStreamImportMode IngressControllerDynamicConfigurationManager InsightsConfig InsightsConfigAPI InsightsOnDemandDataGather InsightsRuntimeExtractor MachineAPIMigration MachineAPIOperatorDisableMachineHealthCheckController MachineAPIProviderOpenStack MachineConfigNodes MaxUnavailableStatefulSet MetricsCollectionProfiles MinimumKubeletVersion MixedCPUsAllocation MultiArchInstallAzure NodeSwap NutanixMultiSubnets OVNObservability OpenShiftPodSecurityAdmission PinnedImages PlatformOperators ProcMountType RouteAdvertisements RouteExternalCertificate ServiceAccountTokenNodeBinding SignatureStores SigstoreImageVerification TranslateStreamCloseWebsocketRequests UpgradeStatus UserNamespacesPodSecurityStandards UserNamespacesSupport VSphereMultiNetworks VolumeAttributesClass VolumeGroupSnapshot]
I0428 12:54:26.262443       1 event.go:377] Event(v1.ObjectReference{Kind:"Node", Namespace:"openshift-machine-config-operator", Name:"ip-10-0-23-133.ec2.internal", UID:"965118d5-7420-4357-a5dd-2adbef9f9dce", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'FeatureGatesInitialized' FeatureGates updated to featuregates.Features{Enabled:[]v1.FeatureGateName{"AWSEFSDriverVolumeMetrics", "AdminNetworkPolicy", "AlibabaPlatform", "AzureWorkloadIdentity", "BareMetalLoadBalancer", "BuildCSIVolumes", "ChunkSizeMiB", "CloudDualStackNodeIPs", "DisableKubeletCloudCredentialProviders", "GCPLabelsTags", "HardwareSpeed", "IngressControllerLBSubnetsAWS", "KMSv1", "ManagedBootImages", "ManagedBootImagesAWS", "MultiArchInstallAWS", "MultiArchInstallGCP", "NetworkDiagnosticsConfig", "NetworkLiveMigration", "NetworkSegmentation", "NewOLM", "NodeDisruptionPolicy", "OnClusterBuild", "PersistentIPsForVirtualization", "PrivateHostedZoneAWS", "SetEIPForNLBIngressController", "StreamingCollectionEncodingToJSON", "StreamingCollectionEncodingToProtobuf", "VSphereControlPlaneMachineSet", "VSphereDriverConfiguration", "VSphereMultiVCenters", "VSphereStaticIPs", "ValidatingAdmissionPolicy"}, Disabled:[]v1.FeatureGateName{"AWSClusterHostedDNS", "AdditionalRoutingCapabilities", "AutomatedEtcdBackup", "BootcNodeManagement", "CSIDriverSharedResource", "ClusterAPIInstall", "ClusterAPIInstallIBMCloud", "ClusterMonitoringConfig", "ConsolePluginContentSecurityPolicy", "DNSNameResolver", "DynamicResourceAllocation", "EtcdBackendQuota", "EventedPLEG", "Example", "ExternalOIDC", "ExternalOIDCWithUIDAndExtraClaimMappings", "GCPClusterHostedDNS", "GatewayAPI", "ImageStreamImportMode", "IngressControllerDynamicConfigurationManager", "InsightsConfig", "InsightsConfigAPI", "InsightsOnDemandDataGather", "InsightsRuntimeExtractor", "MachineAPIMigration", "MachineAPIOperatorDisableMachineHealthCheckController", "MachineAPIProviderOpenStack", "MachineConfigNodes", "MaxUnavailableStatefulSet", "MetricsCollectionProfiles", "MinimumKubeletVersion", "MixedCPUsAllocation", "MultiArchInstallAzure", "NodeSwap", "NutanixMultiSubnets", "OVNObservability", "OpenShiftPodSecurityAdmission", "PinnedImages", "PlatformOperators", "ProcMountType", "RouteAdvertisements", "RouteExternalCertificate", "ServiceAccountTokenNodeBinding", "SignatureStores", "SigstoreImageVerification", "TranslateStreamCloseWebsocketRequests", "UpgradeStatus", "UserNamespacesPodSecurityStandards", "UserNamespacesSupport", "VSphereMultiNetworks", "VolumeAttributesClass", "VolumeGroupSnapshot"}}
I0428 12:54:26.345859       1 reflector.go:368] Caches populated for *v1alpha1.ImageContentSourcePolicy from github.com/openshift/client-go/operator/informers/externalversions/factory.go:125
I0428 12:54:26.348494       1 reflector.go:368] Caches populated for *v1.ImageTagMirrorSet from github.com/openshift/client-go/config/informers/externalversions/factory.go:125
I0428 12:54:26.349634       1 reflector.go:368] Caches populated for *v1.ClusterVersion from github.com/openshift/client-go/config/informers/externalversions/factory.go:125
I0428 12:54:26.350733       1 reflector.go:368] Caches populated for *v1.MachineConfigPool from github.com/openshift/client-go/machineconfiguration/informers/externalversions/factory.go:125
I0428 12:54:26.365291       1 reflector.go:368] Caches populated for *v1.ControllerConfig from github.com/openshift/client-go/machineconfiguration/informers/externalversions/factory.go:125
I0428 12:54:26.374203       1 reflector.go:368] Caches populated for *v1.Secret from k8s.io/client-go/informers/factory.go:160
E0428 12:54:26.374316       1 template_controller.go:245] "Unhandled Error" err="couldn't get ControllerConfig on dependency callback &%!w(errors.StatusError=errors.StatusError{ErrStatus:v1.Status{TypeMeta:v1.TypeMeta{Kind:\"\", APIVersion:\"\"}, ListMeta:v1.ListMeta{SelfLink:\"\", ResourceVersion:\"\", Continue:\"\", RemainingItemCount:(*int64)(nil)}, Status:\"Failure\", Message:\"controllerconfig.machineconfiguration.openshift.io \\\"machine-config-controller\\\" not found\", Reason:\"NotFound\", Details:(*v1.StatusDetails)(0xc000dbfe60), Code:404}})"
I0428 12:54:26.387510       1 template_controller.go:146] Re-syncing ControllerConfig due to secret pull-secret change
I0428 12:54:26.403919       1 reflector.go:368] Caches populated for *v1.Node from k8s.io/client-go/informers/factory.go:160
E0428 12:54:26.404279       1 node_controller.go:535] getting scheduler config failed: cluster scheduler couldn't be found
E0428 12:54:26.404312       1 node_controller.go:535] getting scheduler config failed: cluster scheduler couldn't be found
E0428 12:54:26.404526       1 node_controller.go:535] getting scheduler config failed: cluster scheduler couldn't be found
I0428 12:54:26.466473       1 reflector.go:368] Caches populated for *v1.Node from github.com/openshift/client-go/config/informers/externalversions/factory.go:125
I0428 12:54:26.467641       1 drain_controller.go:169] Starting MachineConfigController-DrainController
I0428 12:54:26.511505       1 reflector.go:368] Caches populated for *v1beta1.MachineSet from github.com/openshift/client-go/machine/informers/externalversions/factory.go:125
I0428 12:54:26.540815       1 reflector.go:368] Caches populated for *v1.ConfigMap from k8s.io/client-go/informers/factory.go:160
I0428 12:54:26.541103       1 machine_set_boot_image_controller.go:235] configMap coreos-bootimages added, reconciling enrolled machine resources
I0428 12:54:26.541118       1 machine_set_boot_image_controller.go:185] MAPI MachineSet clusterpool418-bb45f-xt782-worker-us-east-1a added, reconciling enrolled machine resources
I0428 12:54:26.541123       1 machine_set_boot_image_controller.go:185] MAPI MachineSet clusterpool418-bb45f-xt782-worker-us-east-1b added, reconciling enrolled machine resources
I0428 12:54:26.541146       1 machine_set_boot_image_controller.go:348] MachineConfiguration knobs was not found, so no MAPI machinesets will be enqueued.
I0428 12:54:26.541178       1 machine_set_boot_image_controller.go:348] MachineConfiguration knobs was not found, so no MAPI machinesets will be enqueued.
I0428 12:54:26.541188       1 machine_set_boot_image_controller.go:348] MachineConfiguration knobs was not found, so no MAPI machinesets will be enqueued.
I0428 12:54:26.541192       1 machine_set_boot_image_controller.go:185] MAPI MachineSet clusterpool418-bb45f-xt782-worker-us-east-1c added, reconciling enrolled machine resources
I0428 12:54:26.541196       1 machine_set_boot_image_controller.go:185] MAPI MachineSet clusterpool418-bb45f-xt782-worker-us-east-1d added, reconciling enrolled machine resources
I0428 12:54:26.541202       1 machine_set_boot_image_controller.go:348] MachineConfiguration knobs was not found, so no MAPI machinesets will be enqueued.
I0428 12:54:26.541213       1 machine_set_boot_image_controller.go:348] MachineConfiguration knobs was not found, so no MAPI machinesets will be enqueued.
I0428 12:54:26.541218       1 machine_set_boot_image_controller.go:185] MAPI MachineSet clusterpool418-bb45f-xt782-worker-us-east-1e added, reconciling enrolled machine resources
I0428 12:54:26.541222       1 machine_set_boot_image_controller.go:185] MAPI MachineSet clusterpool418-bb45f-xt782-worker-us-east-1f added, reconciling enrolled machine resources
I0428 12:54:26.541227       1 machine_set_boot_image_controller.go:348] MachineConfiguration knobs was not found, so no MAPI machinesets will be enqueued.
I0428 12:54:26.541235       1 machine_set_boot_image_controller.go:348] MachineConfiguration knobs was not found, so no MAPI machinesets will be enqueued.
I0428 12:54:26.541734       1 reflector.go:368] Caches populated for *v1.Infrastructure from github.com/openshift/client-go/config/informers/externalversions/factory.go:125
I0428 12:54:26.601965       1 reflector.go:368] Caches populated for *v1.KubeletConfig from github.com/openshift/client-go/machineconfiguration/informers/externalversions/factory.go:125
I0428 12:54:26.602377       1 reflector.go:368] Caches populated for *v1.Image from github.com/openshift/client-go/config/informers/externalversions/factory.go:125
I0428 12:54:26.648792       1 reflector.go:368] Caches populated for *v1.ContainerRuntimeConfig from github.com/openshift/client-go/machineconfiguration/informers/externalversions/factory.go:125
I0428 12:54:26.657348       1 reflector.go:368] Caches populated for *v1.MachineConfig from github.com/openshift/client-go/machineconfiguration/informers/externalversions/factory.go:125
I0428 12:54:26.672140       1 template_controller.go:294] Starting MachineConfigController-TemplateController
I0428 12:54:26.686736       1 render_controller.go:141] Starting MachineConfigController-RenderController
I0428 12:54:26.686824       1 container_runtime_config_controller.go:247] Starting MachineConfigController-ContainerRuntimeConfigController
I0428 12:54:26.779367       1 reflector.go:368] Caches populated for *v1.APIServer from github.com/openshift/client-go/config/informers/externalversions/factory.go:125
I0428 12:54:26.779671       1 kubelet_config_controller.go:222] Re-syncing all kubelet config controller generated MachineConfigs due to apiServer cluster change
I0428 12:54:26.779909       1 template_controller.go:198] Re-syncing ControllerConfig due to apiServer cluster change
I0428 12:54:26.798836       1 reflector.go:368] Caches populated for *v1.Scheduler from github.com/openshift/client-go/config/informers/externalversions/factory.go:125
I0428 12:54:26.817302       1 reflector.go:368] Caches populated for *v1.MachineConfiguration from github.com/openshift/client-go/operator/informers/externalversions/factory.go:125
I0428 12:54:26.817709       1 machine_set_boot_image_controller.go:289] Bootimages management configuration has been added, reconciling enrolled machine resources
I0428 12:54:26.817766       1 helpers.go:76] No machine manager were found, so no MAPI machinesets will be enqueued.
I0428 12:54:26.818471       1 machine_set_boot_image_controller.go:380] No MAPI machinesets were enrolled, so no MAPI machinesets will be enqueued.
I0428 12:54:26.857136       1 reflector.go:368] Caches populated for *v1.MachineOSConfig from github.com/openshift/client-go/machineconfiguration/informers/externalversions/factory.go:125
I0428 12:54:26.867825       1 machine_set_boot_image_controller.go:175] Starting MachineConfigController-MachineSetBootImageController
I0428 12:54:26.872266       1 node_controller.go:249] Starting MachineConfigController-NodeController
I0428 12:54:26.872346       1 kubelet_config_controller.go:201] Starting MachineConfigController-KubeletConfigController
I0428 12:54:26.987071       1 reflector.go:368] Caches populated for *v1.Pod from k8s.io/client-go/informers/factory.go:160
I0428 12:54:27.447608       1 kubelet_config_features.go:126] Applied FeatureSet cluster on MachineConfigPool master
I0428 12:54:27.732678       1 reflector.go:368] Caches populated for *v1.ContainerRuntimeConfig from github.com/openshift/client-go/machineconfiguration/informers/externalversions/factory.go:125
I0428 12:54:27.792307       1 kubelet_config_nodes.go:169] Applied Node configuration 97-master-generated-kubelet on MachineConfigPool master
I0428 12:54:28.157561       1 kubelet_config_features.go:126] Applied FeatureSet cluster on MachineConfigPool worker
I0428 12:54:29.152968       1 kubelet_config_nodes.go:169] Applied Node configuration 97-worker-generated-kubelet on MachineConfigPool worker
I0428 12:54:31.179435       1 kubelet_config_features.go:126] Applied FeatureSet cluster on MachineConfigPool master
I0428 12:54:33.446193       1 kubelet_config_features.go:126] Applied FeatureSet cluster on MachineConfigPool worker
I0428 12:58:10.924168       1 render_controller.go:560] Generated machineconfig rendered-master-ea5580a911b3816fcd27c2a4fed8ca94 from 6 configs: [{MachineConfig  00-master  machineconfiguration.openshift.io/v1  } {MachineConfig  01-master-container-runtime  machineconfiguration.openshift.io/v1  } {MachineConfig  01-master-kubelet  machineconfiguration.openshift.io/v1  } {MachineConfig  97-master-generated-kubelet  machineconfiguration.openshift.io/v1  } {MachineConfig  98-master-generated-kubelet  machineconfiguration.openshift.io/v1  } {MachineConfig  99-master-generated-registries  machineconfiguration.openshift.io/v1  }]
I0428 12:58:10.924520       1 event.go:377] Event(v1.ObjectReference{Kind:"MachineConfigPool", Namespace:"openshift-machine-config-operator", Name:"master", UID:"343ec4cd-9bb2-4840-978d-e92f64339321", APIVersion:"machineconfiguration.openshift.io/v1", ResourceVersion:"18617", FieldPath:""}): type: 'Normal' reason: 'RenderedConfigGenerated' rendered-master-ea5580a911b3816fcd27c2a4fed8ca94 successfully generated (release version: 4.18.31, controller version: 9e588a1318823dac89a6bc1790ca584d1100dd57)
I0428 12:58:10.948922       1 render_controller.go:586] Pool master: now targeting: rendered-master-ea5580a911b3816fcd27c2a4fed8ca94
I0428 12:58:10.957177       1 render_controller.go:560] Generated machineconfig rendered-worker-638a7ec4df3c91354114362c96e2308d from 6 configs: [{MachineConfig  00-worker  machineconfiguration.openshift.io/v1  } {MachineConfig  01-worker-container-runtime  machineconfiguration.openshift.io/v1  } {MachineConfig  01-worker-kubelet  machineconfiguration.openshift.io/v1  } {MachineConfig  97-worker-generated-kubelet  machineconfiguration.openshift.io/v1  } {MachineConfig  98-worker-generated-kubelet  machineconfiguration.openshift.io/v1  } {MachineConfig  99-worker-generated-registries  machineconfiguration.openshift.io/v1  }]
I0428 12:58:10.957707       1 event.go:377] Event(v1.ObjectReference{Kind:"MachineConfigPool", Namespace:"openshift-machine-config-operator", Name:"worker", UID:"c5f13e83-dd3a-4bc3-8947-4f02e2a768af", APIVersion:"machineconfiguration.openshift.io/v1", ResourceVersion:"21727", FieldPath:""}): type: 'Normal' reason: 'RenderedConfigGenerated' rendered-worker-638a7ec4df3c91354114362c96e2308d successfully generated (release version: 4.18.31, controller version: 9e588a1318823dac89a6bc1790ca584d1100dd57)
I0428 12:58:10.972279       1 render_controller.go:586] Pool worker: now targeting: rendered-worker-638a7ec4df3c91354114362c96e2308d
I0428 12:58:15.970240       1 node_controller.go:606] Pool master[zone=us-east-1b]: node ip-10-0-46-136.ec2.internal: changed taints
I0428 12:58:16.004474       1 node_controller.go:606] Pool master[zone=us-east-1c]: node ip-10-0-66-149.ec2.internal: changed taints
I0428 12:58:16.027015       1 node_controller.go:596] Pool master: 3 candidate nodes in 3 zones for update, capacity: 1
I0428 12:58:16.027207       1 node_controller.go:1372] Deferring update of machine config operator node: ip-10-0-24-200.ec2.internal
I0428 12:58:16.027224       1 node_controller.go:596] Pool master: filtered to 2 candidate nodes for update, capacity: 1
I0428 12:58:16.027576       1 event.go:377] Event(v1.ObjectReference{Kind:"MachineConfigPool", Namespace:"openshift-machine-config-operator", Name:"master", UID:"343ec4cd-9bb2-4840-978d-e92f64339321", APIVersion:"machineconfiguration.openshift.io/v1", ResourceVersion:"40589", FieldPath:""}): type: 'Normal' reason: 'DeferringOperatorNodeUpdate' Deferring update of machine config operator node ip-10-0-24-200.ec2.internal
I0428 12:58:16.029726       1 node_controller.go:606] Pool master[zone=us-east-1a]: node ip-10-0-24-200.ec2.internal: changed taints
I0428 12:58:16.083443       1 node_controller.go:606] Pool master[zone=us-east-1b]: node ip-10-0-46-136.ec2.internal: changed annotation machineconfiguration.openshift.io/desiredConfig = rendered-master-ea5580a911b3816fcd27c2a4fed8ca94
I0428 12:58:16.089367       1 event.go:377] Event(v1.ObjectReference{Kind:"MachineConfigPool", Namespace:"openshift-machine-config-operator", Name:"master", UID:"343ec4cd-9bb2-4840-978d-e92f64339321", APIVersion:"machineconfiguration.openshift.io/v1", ResourceVersion:"40589", FieldPath:""}): type: 'Normal' reason: 'AnnotationChange' Node ip-10-0-46-136.ec2.internal now has machineconfiguration.openshift.io/desiredConfig=rendered-master-ea5580a911b3816fcd27c2a4fed8ca94
I0428 12:58:16.092439       1 event.go:377] Event(v1.ObjectReference{Kind:"MachineConfigPool", Namespace:"openshift-machine-config-operator", Name:"master", UID:"343ec4cd-9bb2-4840-978d-e92f64339321", APIVersion:"machineconfiguration.openshift.io/v1", ResourceVersion:"40589", FieldPath:""}): type: 'Normal' reason: 'SetDesiredConfig' Targeted node ip-10-0-46-136.ec2.internal to MachineConfig: rendered-master-ea5580a911b3816fcd27c2a4fed8ca94
I0428 12:58:16.166336       1 node_controller.go:606] Pool worker[zone=us-east-1a]: node ip-10-0-23-133.ec2.internal: changed taints
I0428 12:58:16.961628       1 node_controller.go:606] Pool worker[zone=us-east-1b]: node ip-10-0-55-209.ec2.internal: changed taints
I0428 12:58:17.130509       1 node_controller.go:606] Pool master[zone=us-east-1b]: node ip-10-0-46-136.ec2.internal: changed annotation machineconfiguration.openshift.io/state = Working
I0428 12:58:17.130755       1 event.go:377] Event(v1.ObjectReference{Kind:"MachineConfigPool", Namespace:"openshift-machine-config-operator", Name:"master", UID:"343ec4cd-9bb2-4840-978d-e92f64339321", APIVersion:"machineconfiguration.openshift.io/v1", ResourceVersion:"40893", FieldPath:""}): type: 'Normal' reason: 'AnnotationChange' Node ip-10-0-46-136.ec2.internal now has machineconfiguration.openshift.io/state=Working
I0428 12:58:17.569758       1 node_controller.go:596] Pool worker: 3 candidate nodes in 3 zones for update, capacity: 1
I0428 12:58:17.570204       1 node_controller.go:606] Pool worker[zone=us-east-1c]: node ip-10-0-89-36.ec2.internal: changed taints
I0428 12:58:17.961699       1 event.go:377] Event(v1.ObjectReference{Kind:"MachineConfigPool", Namespace:"openshift-machine-config-operator", Name:"worker", UID:"c5f13e83-dd3a-4bc3-8947-4f02e2a768af", APIVersion:"machineconfiguration.openshift.io/v1", ResourceVersion:"40594", FieldPath:""}): type: 'Normal' reason: 'SetDesiredConfig' Targeted node ip-10-0-23-133.ec2.internal to MachineConfig: rendered-worker-638a7ec4df3c91354114362c96e2308d
I0428 12:58:17.962009       1 node_controller.go:606] Pool worker[zone=us-east-1a]: node ip-10-0-23-133.ec2.internal: changed annotation machineconfiguration.openshift.io/desiredConfig = rendered-worker-638a7ec4df3c91354114362c96e2308d
I0428 12:58:19.089475       1 node_controller.go:606] Pool worker[zone=us-east-1a]: node ip-10-0-23-133.ec2.internal: changed annotation machineconfiguration.openshift.io/state = Working
I0428 12:58:21.003131       1 node_controller.go:606] Pool master[zone=us-east-1b]: node ip-10-0-46-136.ec2.internal: changed taints
I0428 12:58:21.211684       1 node_controller.go:606] Pool worker[zone=us-east-1a]: node ip-10-0-23-133.ec2.internal: changed taints
I0428 12:58:34.852451       1 drain_controller.go:183] node ip-10-0-23-133.ec2.internal: uncordoning
I0428 12:58:34.852493       1 drain_controller.go:183] node ip-10-0-23-133.ec2.internal: initiating uncordon (currently schedulable: true)
I0428 12:58:34.856047       1 drain_controller.go:183] node ip-10-0-23-133.ec2.internal: uncordon succeeded (currently schedulable: true)
I0428 12:58:34.856065       1 drain_controller.go:183] node ip-10-0-23-133.ec2.internal: operation successful; applying completion annotation
I0428 12:58:37.404211       1 drain_controller.go:183] node ip-10-0-46-136.ec2.internal: uncordoning
I0428 12:58:37.404246       1 drain_controller.go:183] node ip-10-0-46-136.ec2.internal: initiating uncordon (currently schedulable: true)
I0428 12:58:37.411252       1 drain_controller.go:183] node ip-10-0-46-136.ec2.internal: uncordon succeeded (currently schedulable: true)
I0428 12:58:37.412417       1 drain_controller.go:183] node ip-10-0-46-136.ec2.internal: operation successful; applying completion annotation
I0428 12:58:39.868074       1 node_controller.go:606] Pool worker[zone=us-east-1a]: node ip-10-0-23-133.ec2.internal: Completed update to rendered-worker-638a7ec4df3c91354114362c96e2308d
I0428 12:58:42.477131       1 node_controller.go:606] Pool master[zone=us-east-1b]: node ip-10-0-46-136.ec2.internal: Completed update to rendered-master-ea5580a911b3816fcd27c2a4fed8ca94
I0428 12:58:44.871530       1 node_controller.go:596] Pool worker: 2 candidate nodes in 2 zones for update, capacity: 1
I0428 12:58:44.889018       1 event.go:377] Event(v1.ObjectReference{Kind:"MachineConfigPool", Namespace:"openshift-machine-config-operator", Name:"worker", UID:"c5f13e83-dd3a-4bc3-8947-4f02e2a768af", APIVersion:"machineconfiguration.openshift.io/v1", ResourceVersion:"41126", FieldPath:""}): type: 'Normal' reason: 'SetDesiredConfig' Targeted node ip-10-0-55-209.ec2.internal to MachineConfig: rendered-worker-638a7ec4df3c91354114362c96e2308d
I0428 12:58:44.891281       1 node_controller.go:606] Pool worker[zone=us-east-1b]: node ip-10-0-55-209.ec2.internal: changed annotation machineconfiguration.openshift.io/desiredConfig = rendered-worker-638a7ec4df3c91354114362c96e2308d
I0428 12:58:46.044390       1 node_controller.go:606] Pool worker[zone=us-east-1b]: node ip-10-0-55-209.ec2.internal: changed annotation machineconfiguration.openshift.io/state = Working
I0428 12:58:47.484190       1 node_controller.go:596] Pool master: 2 candidate nodes in 2 zones for update, capacity: 1
I0428 12:58:47.484337       1 node_controller.go:1372] Deferring update of machine config operator node: ip-10-0-24-200.ec2.internal
I0428 12:58:47.484347       1 node_controller.go:596] Pool master: filtered to 1 candidate nodes for update, capacity: 1
I0428 12:58:47.484499       1 event.go:377] Event(v1.ObjectReference{Kind:"MachineConfigPool", Namespace:"openshift-machine-config-operator", Name:"master", UID:"343ec4cd-9bb2-4840-978d-e92f64339321", APIVersion:"machineconfiguration.openshift.io/v1", ResourceVersion:"40893", FieldPath:""}): type: 'Normal' reason: 'DeferringOperatorNodeUpdate' Deferring update of machine config operator node ip-10-0-24-200.ec2.internal
I0428 12:58:47.506412       1 event.go:377] Event(v1.ObjectReference{Kind:"MachineConfigPool", Namespace:"openshift-machine-config-operator", Name:"master", UID:"343ec4cd-9bb2-4840-978d-e92f64339321", APIVersion:"machineconfiguration.openshift.io/v1", ResourceVersion:"40893", FieldPath:""}): type: 'Normal' reason: 'SetDesiredConfig' Targeted node ip-10-0-66-149.ec2.internal to MachineConfig: rendered-master-ea5580a911b3816fcd27c2a4fed8ca94
I0428 12:58:47.517757       1 node_controller.go:606] Pool master[zone=us-east-1c]: node ip-10-0-66-149.ec2.internal: changed annotation machineconfiguration.openshift.io/desiredConfig = rendered-master-ea5580a911b3816fcd27c2a4fed8ca94
I0428 12:58:47.518010       1 event.go:377] Event(v1.ObjectReference{Kind:"MachineConfigPool", Namespace:"openshift-machine-config-operator", Name:"master", UID:"343ec4cd-9bb2-4840-978d-e92f64339321", APIVersion:"machineconfiguration.openshift.io/v1", ResourceVersion:"40893", FieldPath:""}): type: 'Normal' reason: 'AnnotationChange' Node ip-10-0-66-149.ec2.internal now has machineconfiguration.openshift.io/desiredConfig=rendered-master-ea5580a911b3816fcd27c2a4fed8ca94
I0428 12:58:49.187489       1 node_controller.go:606] Pool master[zone=us-east-1c]: node ip-10-0-66-149.ec2.internal: changed annotation machineconfiguration.openshift.io/state = Working
I0428 12:58:49.190878       1 event.go:377] Event(v1.ObjectReference{Kind:"MachineConfigPool", Namespace:"openshift-machine-config-operator", Name:"master", UID:"343ec4cd-9bb2-4840-978d-e92f64339321", APIVersion:"machineconfiguration.openshift.io/v1", ResourceVersion:"42573", FieldPath:""}): type: 'Normal' reason: 'AnnotationChange' Node ip-10-0-66-149.ec2.internal now has machineconfiguration.openshift.io/state=Working
I0428 12:58:49.920412       1 node_controller.go:606] Pool worker[zone=us-east-1b]: node ip-10-0-55-209.ec2.internal: changed taints
E0428 12:58:50.056168       1 render_controller.go:464] Error syncing Generated MCFG: Operation cannot be fulfilled on machineconfigpools.machineconfiguration.openshift.io "worker": the object has been modified; please apply your changes to the latest version and try again
E0428 12:58:50.064453       1 render_controller.go:486] Error updating MachineConfigPool worker: Operation cannot be fulfilled on machineconfigpools.machineconfiguration.openshift.io "worker": the object has been modified; please apply your changes to the latest version and try again
I0428 12:58:50.064488       1 render_controller.go:394] Error syncing machineconfigpool worker: Operation cannot be fulfilled on machineconfigpools.machineconfiguration.openshift.io "worker": the object has been modified; please apply your changes to the latest version and try again
I0428 12:58:52.586969       1 node_controller.go:606] Pool master[zone=us-east-1c]: node ip-10-0-66-149.ec2.internal: changed taints
E0428 12:58:53.032224       1 render_controller.go:464] Error syncing Generated MCFG: Operation cannot be fulfilled on machineconfigpools.machineconfiguration.openshift.io "master": the object has been modified; please apply your changes to the latest version and try again
E0428 12:58:53.037980       1 render_controller.go:486] Error updating MachineConfigPool master: Operation cannot be fulfilled on machineconfigpools.machineconfiguration.openshift.io "master": the object has been modified; please apply your changes to the latest version and try again
I0428 12:58:53.038005       1 render_controller.go:394] Error syncing machineconfigpool master: Operation cannot be fulfilled on machineconfigpools.machineconfiguration.openshift.io "master": the object has been modified; please apply your changes to the latest version and try again
I0428 12:59:01.947672       1 drain_controller.go:183] node ip-10-0-55-209.ec2.internal: uncordoning
I0428 12:59:01.947719       1 drain_controller.go:183] node ip-10-0-55-209.ec2.internal: initiating uncordon (currently schedulable: true)
I0428 12:59:01.958425       1 drain_controller.go:183] node ip-10-0-55-209.ec2.internal: uncordon succeeded (currently schedulable: true)
I0428 12:59:01.958453       1 drain_controller.go:183] node ip-10-0-55-209.ec2.internal: operation successful; applying completion annotation
I0428 12:59:06.970982       1 node_controller.go:606] Pool worker[zone=us-east-1b]: node ip-10-0-55-209.ec2.internal: Completed update to rendered-worker-638a7ec4df3c91354114362c96e2308d
I0428 12:59:11.975341       1 node_controller.go:596] Pool worker: 1 candidate nodes in 1 zones for update, capacity: 1
I0428 12:59:11.999366       1 event.go:377] Event(v1.ObjectReference{Kind:"MachineConfigPool", Namespace:"openshift-machine-config-operator", Name:"worker", UID:"c5f13e83-dd3a-4bc3-8947-4f02e2a768af", APIVersion:"machineconfiguration.openshift.io/v1", ResourceVersion:"42729", FieldPath:""}): type: 'Normal' reason: 'SetDesiredConfig' Targeted node ip-10-0-89-36.ec2.internal to MachineConfig: rendered-worker-638a7ec4df3c91354114362c96e2308d
I0428 12:59:12.001867       1 node_controller.go:606] Pool worker[zone=us-east-1c]: node ip-10-0-89-36.ec2.internal: changed annotation machineconfiguration.openshift.io/desiredConfig = rendered-worker-638a7ec4df3c91354114362c96e2308d
I0428 12:59:13.131485       1 node_controller.go:606] Pool worker[zone=us-east-1c]: node ip-10-0-89-36.ec2.internal: changed annotation machineconfiguration.openshift.io/state = Working
I0428 12:59:17.050543       1 node_controller.go:606] Pool worker[zone=us-east-1c]: node ip-10-0-89-36.ec2.internal: changed taints
E0428 12:59:17.242441       1 render_controller.go:464] Error syncing Generated MCFG: Operation cannot be fulfilled on machineconfigpools.machineconfiguration.openshift.io "worker": the object has been modified; please apply your changes to the latest version and try again
E0428 12:59:17.247636       1 render_controller.go:486] Error updating MachineConfigPool worker: Operation cannot be fulfilled on machineconfigpools.machineconfiguration.openshift.io "worker": the object has been modified; please apply your changes to the latest version and try again
I0428 12:59:17.247716       1 render_controller.go:394] Error syncing machineconfigpool worker: Operation cannot be fulfilled on machineconfigpools.machineconfiguration.openshift.io "worker": the object has been modified; please apply your changes to the latest version and try again
I0428 12:59:26.567750       1 drain_controller.go:183] node ip-10-0-89-36.ec2.internal: uncordoning
I0428 12:59:26.567789       1 drain_controller.go:183] node ip-10-0-89-36.ec2.internal: initiating uncordon (currently schedulable: true)
I0428 12:59:26.583651       1 drain_controller.go:183] node ip-10-0-89-36.ec2.internal: uncordon succeeded (currently schedulable: true)
I0428 12:59:26.583669       1 drain_controller.go:183] node ip-10-0-89-36.ec2.internal: operation successful; applying completion annotation
I0428 12:59:31.578642       1 node_controller.go:606] Pool worker[zone=us-east-1c]: node ip-10-0-89-36.ec2.internal: Completed update to rendered-worker-638a7ec4df3c91354114362c96e2308d
I0428 12:59:36.748511       1 status.go:267] Pool worker: All nodes are updated with MachineConfig rendered-worker-638a7ec4df3c91354114362c96e2308d
I0428 12:59:54.324042       1 drain_controller.go:183] node ip-10-0-66-149.ec2.internal: uncordoning
I0428 12:59:54.324175       1 drain_controller.go:183] node ip-10-0-66-149.ec2.internal: initiating uncordon (currently schedulable: true)
I0428 12:59:54.327765       1 drain_controller.go:183] node ip-10-0-66-149.ec2.internal: uncordon succeeded (currently schedulable: true)
I0428 12:59:54.327847       1 drain_controller.go:183] node ip-10-0-66-149.ec2.internal: operation successful; applying completion annotation
I0428 12:59:59.422370       1 node_controller.go:606] Pool master[zone=us-east-1c]: node ip-10-0-66-149.ec2.internal: Completed update to rendered-master-ea5580a911b3816fcd27c2a4fed8ca94
I0428 13:00:04.427659       1 node_controller.go:596] Pool master: 1 candidate nodes in 1 zones for update, capacity: 1
I0428 13:00:04.427678       1 node_controller.go:596] Pool master: filtered to 1 candidate nodes for update, capacity: 1
I0428 13:00:04.442573       1 event.go:377] Event(v1.ObjectReference{Kind:"MachineConfigPool", Namespace:"openshift-machine-config-operator", Name:"master", UID:"343ec4cd-9bb2-4840-978d-e92f64339321", APIVersion:"machineconfiguration.openshift.io/v1", ResourceVersion:"42886", FieldPath:""}): type: 'Normal' reason: 'SetDesiredConfig' Targeted node ip-10-0-24-200.ec2.internal to MachineConfig: rendered-master-ea5580a911b3816fcd27c2a4fed8ca94
I0428 13:00:04.448323       1 node_controller.go:606] Pool master[zone=us-east-1a]: node ip-10-0-24-200.ec2.internal: changed annotation machineconfiguration.openshift.io/desiredConfig = rendered-master-ea5580a911b3816fcd27c2a4fed8ca94
I0428 13:00:04.448471       1 event.go:377] Event(v1.ObjectReference{Kind:"MachineConfigPool", Namespace:"openshift-machine-config-operator", Name:"master", UID:"343ec4cd-9bb2-4840-978d-e92f64339321", APIVersion:"machineconfiguration.openshift.io/v1", ResourceVersion:"42886", FieldPath:""}): type: 'Normal' reason: 'AnnotationChange' Node ip-10-0-24-200.ec2.internal now has machineconfiguration.openshift.io/desiredConfig=rendered-master-ea5580a911b3816fcd27c2a4fed8ca94
I0428 13:00:06.005676       1 node_controller.go:606] Pool master[zone=us-east-1a]: node ip-10-0-24-200.ec2.internal: changed annotation machineconfiguration.openshift.io/state = Working
I0428 13:00:06.005863       1 event.go:377] Event(v1.ObjectReference{Kind:"MachineConfigPool", Namespace:"openshift-machine-config-operator", Name:"master", UID:"343ec4cd-9bb2-4840-978d-e92f64339321", APIVersion:"machineconfiguration.openshift.io/v1", ResourceVersion:"47583", FieldPath:""}): type: 'Normal' reason: 'AnnotationChange' Node ip-10-0-24-200.ec2.internal now has machineconfiguration.openshift.io/state=Working
I0428 13:00:09.496489       1 node_controller.go:606] Pool master[zone=us-east-1a]: node ip-10-0-24-200.ec2.internal: changed taints
E0428 13:00:09.688533       1 render_controller.go:464] Error syncing Generated MCFG: Operation cannot be fulfilled on machineconfigpools.machineconfiguration.openshift.io "master": the object has been modified; please apply your changes to the latest version and try again
E0428 13:00:09.695830       1 render_controller.go:486] Error updating MachineConfigPool master: Operation cannot be fulfilled on machineconfigpools.machineconfiguration.openshift.io "master": the object has been modified; please apply your changes to the latest version and try again
I0428 13:00:09.695937       1 render_controller.go:394] Error syncing machineconfigpool master: Operation cannot be fulfilled on machineconfigpools.machineconfiguration.openshift.io "master": the object has been modified; please apply your changes to the latest version and try again
I0428 13:00:35.944264       1 drain_controller.go:183] node ip-10-0-24-200.ec2.internal: uncordoning
I0428 13:00:35.944305       1 drain_controller.go:183] node ip-10-0-24-200.ec2.internal: initiating uncordon (currently schedulable: true)
I0428 13:00:35.948713       1 drain_controller.go:183] node ip-10-0-24-200.ec2.internal: uncordon succeeded (currently schedulable: true)
I0428 13:00:35.948731       1 drain_controller.go:183] node ip-10-0-24-200.ec2.internal: operation successful; applying completion annotation
I0428 13:00:40.976117       1 node_controller.go:606] Pool master[zone=us-east-1a]: node ip-10-0-24-200.ec2.internal: Completed update to rendered-master-ea5580a911b3816fcd27c2a4fed8ca94
I0428 13:00:45.984105       1 status.go:267] Pool master: All nodes are updated with MachineConfig rendered-master-ea5580a911b3816fcd27c2a4fed8ca94