I0128 12:56:19.970840 1 start.go:62] Version: 7cb990c2 (6fd31b7e48fefb01d263cd19b733aca57ba2cf95) I0128 12:56:19.971252 1 leaderelection.go:121] The leader election gives 4 retries and allows for 30s of clock skew. The kube-apiserver downtime tolerance is 78s. Worst non-graceful lease acquisition is 2m43s. Worst graceful lease acquisition is {26s}. I0128 12:56:19.971349 1 envvar.go:172] "Feature gate default state" feature="InformerResourceVersion" enabled=false I0128 12:56:19.971376 1 envvar.go:172] "Feature gate default state" feature="WatchListClient" enabled=false I0128 12:56:19.971395 1 envvar.go:172] "Feature gate default state" feature="ClientsAllowCBOR" enabled=false I0128 12:56:19.971412 1 envvar.go:172] "Feature gate default state" feature="ClientsPreferCBOR" enabled=false I0128 12:56:20.000937 1 leaderelection.go:257] attempting to acquire leader lease openshift-machine-config-operator/machine-config-controller... I0128 12:59:08.652665 1 leaderelection.go:271] successfully acquired lease openshift-machine-config-operator/machine-config-controller I0128 12:59:08.660046 1 simple_featuregate_reader.go:171] Starting feature-gate-detector I0128 12:59:08.660075 1 metrics.go:92] Registering Prometheus metrics I0128 12:59:08.660129 1 metrics.go:99] Starting metrics listener on 127.0.0.1:8797 I0128 12:59:08.669579 1 certrotation_controller.go:170] Adding MCS CA/TLS cert rotator I0128 12:59:08.673915 1 reflector.go:376] Caches populated for *v1.Infrastructure from github.com/openshift/client-go/config/informers/externalversions/factory.go:125 I0128 12:59:08.677637 1 reflector.go:376] Caches populated for *v1.ImageTagMirrorSet from github.com/openshift/client-go/config/informers/externalversions/factory.go:125 I0128 12:59:08.677992 1 reflector.go:376] Caches populated for *v1.Secret from k8s.io/client-go/informers/factory.go:160 I0128 12:59:08.678633 1 reflector.go:376] Caches populated for *v1.APIServer from github.com/openshift/client-go/config/informers/externalversions/factory.go:125 I0128 12:59:08.679455 1 reflector.go:376] Caches populated for *v1.Node from github.com/openshift/client-go/config/informers/externalversions/factory.go:125 I0128 12:59:08.680107 1 reflector.go:376] Caches populated for *v1.FeatureGate from github.com/openshift/client-go/config/informers/externalversions/factory.go:125 I0128 12:59:08.680408 1 reflector.go:376] Caches populated for *v1.ConfigMap from k8s.io/client-go/informers/factory.go:160 I0128 12:59:08.680836 1 reflector.go:376] Caches populated for *v1.Image from github.com/openshift/client-go/config/informers/externalversions/factory.go:125 I0128 12:59:08.681213 1 reflector.go:376] Caches populated for *v1.Secret from k8s.io/client-go/informers/factory.go:160 I0128 12:59:08.681451 1 reflector.go:376] Caches populated for *v1alpha1.ImageContentSourcePolicy from github.com/openshift/client-go/operator/informers/externalversions/factory.go:125 I0128 12:59:08.682182 1 kubelet_config_controller.go:222] Re-syncing all kubelet config controller generated MachineConfigs due to apiServer cluster change E0128 12:59:08.684451 1 template_controller.go:245] "Unhandled Error" err="couldn't get ControllerConfig on dependency callback &%!w(errors.StatusError=errors.StatusError{ErrStatus:v1.Status{TypeMeta:v1.TypeMeta{Kind:\"\", APIVersion:\"\"}, ListMeta:v1.ListMeta{SelfLink:\"\", ResourceVersion:\"\", Continue:\"\", RemainingItemCount:(*int64)(nil)}, Status:\"Failure\", Message:\"controllerconfig.machineconfiguration.openshift.io \\\"machine-config-controller\\\" not found\", Reason:\"NotFound\", Details:(*v1.StatusDetails)(0xc0006f5e00), Code:404}})" E0128 12:59:08.684559 1 template_controller.go:245] "Unhandled Error" err="couldn't get ControllerConfig on dependency callback &%!w(errors.StatusError=errors.StatusError{ErrStatus:v1.Status{TypeMeta:v1.TypeMeta{Kind:\"\", APIVersion:\"\"}, ListMeta:v1.ListMeta{SelfLink:\"\", ResourceVersion:\"\", Continue:\"\", RemainingItemCount:(*int64)(nil)}, Status:\"Failure\", Message:\"controllerconfig.machineconfiguration.openshift.io \\\"machine-config-controller\\\" not found\", Reason:\"NotFound\", Details:(*v1.StatusDetails)(0xc000f062a0), Code:404}})" I0128 12:59:08.687178 1 reflector.go:376] Caches populated for *v1.ImageDigestMirrorSet from github.com/openshift/client-go/config/informers/externalversions/factory.go:125 I0128 12:59:08.689222 1 reflector.go:376] Caches populated for *v1.Secret from k8s.io/client-go/informers/factory.go:160 E0128 12:59:08.690841 1 node_controller.go:536] getting scheduler config failed: cluster scheduler couldn't be found E0128 12:59:08.690914 1 node_controller.go:536] getting scheduler config failed: cluster scheduler couldn't be found E0128 12:59:08.690931 1 node_controller.go:536] getting scheduler config failed: cluster scheduler couldn't be found I0128 12:59:08.690401 1 reflector.go:376] Caches populated for *v1.Node from k8s.io/client-go/informers/factory.go:160 I0128 12:59:08.691247 1 event.go:377] Event(v1.ObjectReference{Kind:"Node", Namespace:"openshift-machine-config-operator", Name:"ip-10-0-21-80.ec2.internal", UID:"4f03251c-0a4d-4d5a-9cf7-c46320dd83e7", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'FeatureGatesInitialized' FeatureGates updated to featuregates.Features{Enabled:[]v1.FeatureGateName{"AWSEFSDriverVolumeMetrics", "AdditionalRoutingCapabilities", "AdminNetworkPolicy", "AlibabaPlatform", "AzureWorkloadIdentity", "BareMetalLoadBalancer", "BuildCSIVolumes", "CPMSMachineNamePrefix", "ChunkSizeMiB", "CloudDualStackNodeIPs", "ConsolePluginContentSecurityPolicy", "DisableKubeletCloudCredentialProviders", "GCPLabelsTags", "GatewayAPI", "GatewayAPIController", "HardwareSpeed", "IngressControllerLBSubnetsAWS", "KMSv1", "MachineConfigNodes", "ManagedBootImages", "ManagedBootImagesAWS", "MetricsCollectionProfiles", "MultiArchInstallAWS", "MultiArchInstallGCP", "NetworkDiagnosticsConfig", "NetworkLiveMigration", "NetworkSegmentation", "NewOLM", "NodeDisruptionPolicy", "OnClusterBuild", "PersistentIPsForVirtualization", "PinnedImages", "PrivateHostedZoneAWS", "RouteAdvertisements", "RouteExternalCertificate", "ServiceAccountTokenNodeBinding", "SetEIPForNLBIngressController", "VSphereDriverConfiguration", "VSphereMultiVCenters", "ValidatingAdmissionPolicy"}, Disabled:[]v1.FeatureGateName{"AWSClusterHostedDNS", "AutomatedEtcdBackup", "BootcNodeManagement", "ClusterAPIInstall", "ClusterAPIInstallIBMCloud", "ClusterMonitoringConfig", "ClusterVersionOperatorConfiguration", "DNSNameResolver", "DualReplica", "DyanmicServiceEndpointIBMCloud", "DynamicResourceAllocation", "EtcdBackendQuota", "EventedPLEG", "Example", "Example2", "ExternalOIDC", "ExternalOIDCWithUIDAndExtraClaimMappings", "GCPClusterHostedDNS", "GCPCustomAPIEndpoints", "HighlyAvailableArbiter", "ImageStreamImportMode", "IngressControllerDynamicConfigurationManager", "InsightsConfig", "InsightsConfigAPI", "InsightsOnDemandDataGather", "InsightsRuntimeExtractor", "KMSEncryptionProvider", "MachineAPIMigration", "MachineAPIOperatorDisableMachineHealthCheckController", "MachineAPIProviderOpenStack", "MaxUnavailableStatefulSet", "MinimumKubeletVersion", "MixedCPUsAllocation", "MultiArchInstallAzure", "NewOLMCatalogdAPIV1Metas", "NewOLMOwnSingleNamespace", "NewOLMPreflightPermissionChecks", "NodeSwap", "NutanixMultiSubnets", "OVNObservability", "OpenShiftPodSecurityAdmission", "PlatformOperators", "ProcMountType", "SELinuxChangePolicy", "SELinuxMount", "ShortCertRotation", "SignatureStores", "SigstoreImageVerification", "SigstoreImageVerificationPKI", "StreamingCollectionEncodingToJSON", "StreamingCollectionEncodingToProtobuf", "TranslateStreamCloseWebsocketRequests", "UpgradeStatus", "UserNamespacesPodSecurityStandards", "UserNamespacesSupport", "VSphereConfigurableMaxAllowedBlockVolumesPerNode", "VSphereHostVMGroupZonal", "VSphereMultiDisk", "VSphereMultiNetworks", "VolumeAttributesClass", "VolumeGroupSnapshot"}} I0128 12:59:08.691529 1 template_controller.go:198] Re-syncing ControllerConfig due to apiServer cluster change I0128 12:59:08.691563 1 start.go:121] FeatureGates initialized: enabled=[AWSEFSDriverVolumeMetrics AdditionalRoutingCapabilities AdminNetworkPolicy AlibabaPlatform AzureWorkloadIdentity BareMetalLoadBalancer BuildCSIVolumes CPMSMachineNamePrefix ChunkSizeMiB CloudDualStackNodeIPs ConsolePluginContentSecurityPolicy DisableKubeletCloudCredentialProviders GCPLabelsTags GatewayAPI GatewayAPIController HardwareSpeed IngressControllerLBSubnetsAWS KMSv1 MachineConfigNodes ManagedBootImages ManagedBootImagesAWS MetricsCollectionProfiles MultiArchInstallAWS MultiArchInstallGCP NetworkDiagnosticsConfig NetworkLiveMigration NetworkSegmentation NewOLM NodeDisruptionPolicy OnClusterBuild PersistentIPsForVirtualization PinnedImages PrivateHostedZoneAWS RouteAdvertisements RouteExternalCertificate ServiceAccountTokenNodeBinding SetEIPForNLBIngressController VSphereDriverConfiguration VSphereMultiVCenters ValidatingAdmissionPolicy] disabled=[AWSClusterHostedDNS AutomatedEtcdBackup BootcNodeManagement ClusterAPIInstall ClusterAPIInstallIBMCloud ClusterMonitoringConfig ClusterVersionOperatorConfiguration DNSNameResolver DualReplica DyanmicServiceEndpointIBMCloud DynamicResourceAllocation EtcdBackendQuota EventedPLEG Example Example2 ExternalOIDC ExternalOIDCWithUIDAndExtraClaimMappings GCPClusterHostedDNS GCPCustomAPIEndpoints HighlyAvailableArbiter ImageStreamImportMode IngressControllerDynamicConfigurationManager InsightsConfig InsightsConfigAPI InsightsOnDemandDataGather InsightsRuntimeExtractor KMSEncryptionProvider MachineAPIMigration MachineAPIOperatorDisableMachineHealthCheckController MachineAPIProviderOpenStack MaxUnavailableStatefulSet MinimumKubeletVersion MixedCPUsAllocation MultiArchInstallAzure NewOLMCatalogdAPIV1Metas NewOLMOwnSingleNamespace NewOLMPreflightPermissionChecks NodeSwap NutanixMultiSubnets OVNObservability OpenShiftPodSecurityAdmission PlatformOperators ProcMountType SELinuxChangePolicy SELinuxMount ShortCertRotation SignatureStores SigstoreImageVerification SigstoreImageVerificationPKI StreamingCollectionEncodingToJSON StreamingCollectionEncodingToProtobuf TranslateStreamCloseWebsocketRequests UpgradeStatus UserNamespacesPodSecurityStandards UserNamespacesSupport VSphereConfigurableMaxAllowedBlockVolumesPerNode VSphereHostVMGroupZonal VSphereMultiDisk VSphereMultiNetworks VolumeAttributesClass VolumeGroupSnapshot] I0128 12:59:08.692741 1 reflector.go:376] Caches populated for *v1.ClusterVersion from github.com/openshift/client-go/config/informers/externalversions/factory.go:125 I0128 12:59:08.696782 1 template_controller.go:146] Re-syncing ControllerConfig due to secret pull-secret change I0128 12:59:08.703421 1 certrotation_controller.go:192] Starting machineconfigcontroller-certrotationcontroller I0128 12:59:08.703461 1 certrotation_controller.go:180] Waiting for machineconfigcontroller-certrotationcontroller I0128 12:59:08.703471 1 certrotation_controller.go:188] Finished waiting for machineconfigcontroller-certrotationcontroller I0128 12:59:08.703714 1 machine_set_boot_image_controller.go:233] configMap coreos-bootimages added, reconciling enrolled machine resources E0128 12:59:08.704167 1 machine_set_boot_image_controller.go:347] MachineConfiguration/cluster has not been created yet I0128 12:59:08.710265 1 reflector.go:376] Caches populated for *v1beta1.MachineSet from github.com/openshift/client-go/machine/informers/externalversions/factory.go:125 I0128 12:59:08.710848 1 machine_set_boot_image_controller.go:183] MAPI MachineSet clusterpool419-z4ngz-swxvh-worker-us-east-1a added, reconciling enrolled machine resources I0128 12:59:08.710903 1 machine_set_boot_image_controller.go:183] MAPI MachineSet clusterpool419-z4ngz-swxvh-worker-us-east-1b added, reconciling enrolled machine resources I0128 12:59:08.711031 1 machine_set_boot_image_controller.go:183] MAPI MachineSet clusterpool419-z4ngz-swxvh-worker-us-east-1c added, reconciling enrolled machine resources I0128 12:59:08.711076 1 machine_set_boot_image_controller.go:183] MAPI MachineSet clusterpool419-z4ngz-swxvh-worker-us-east-1d added, reconciling enrolled machine resources I0128 12:59:08.711096 1 machine_set_boot_image_controller.go:183] MAPI MachineSet clusterpool419-z4ngz-swxvh-worker-us-east-1e added, reconciling enrolled machine resources I0128 12:59:08.711116 1 machine_set_boot_image_controller.go:183] MAPI MachineSet clusterpool419-z4ngz-swxvh-worker-us-east-1f added, reconciling enrolled machine resources I0128 12:59:08.715406 1 hostnames.go:43] syncing hostnames: [api-int.clusterpool419-z4ngz.rhtap.devcluster.openshift.com] I0128 12:59:08.715489 1 base_controller.go:76] Waiting for caches to sync for CertRotationController I0128 12:59:08.715533 1 base_controller.go:82] Caches are synced for CertRotationController I0128 12:59:08.715561 1 base_controller.go:119] Starting #1 worker of CertRotationController controller ... I0128 12:59:08.715616 1 certrotation_controller.go:334] configMap machine-config-server-ca added, reconciling all user data secrets I0128 12:59:08.715729 1 certrotation_controller.go:373] secret worker-user-data added, ensuring CA is up to date I0128 12:59:08.715921 1 hostnames.go:43] syncing hostnames: [api-int.clusterpool419-z4ngz.rhtap.devcluster.openshift.com] I0128 12:59:08.716682 1 certrotation_controller.go:373] secret master-user-data added, ensuring CA is up to date I0128 12:59:08.735241 1 reflector.go:376] Caches populated for *v1.Pod from k8s.io/client-go/informers/factory.go:160 I0128 12:59:08.765297 1 reflector.go:376] Caches populated for *v1.ControllerConfig from github.com/openshift/client-go/machineconfiguration/informers/externalversions/factory.go:125 I0128 12:59:08.767561 1 reflector.go:376] Caches populated for *v1.KubeletConfig from github.com/openshift/client-go/machineconfiguration/informers/externalversions/factory.go:125 I0128 12:59:08.778197 1 reflector.go:376] Caches populated for *v1.Scheduler from github.com/openshift/client-go/config/informers/externalversions/factory.go:125 I0128 12:59:08.799293 1 reflector.go:376] Caches populated for *v1.MachineConfigPool from github.com/openshift/client-go/machineconfiguration/informers/externalversions/factory.go:125 I0128 12:59:08.801471 1 reflector.go:376] Caches populated for *v1.ContainerRuntimeConfig from github.com/openshift/client-go/machineconfiguration/informers/externalversions/factory.go:125 I0128 12:59:08.804445 1 drain_controller.go:179] Starting MachineConfigController-DrainController I0128 12:59:08.804472 1 kubelet_config_controller.go:201] Starting MachineConfigController-KubeletConfigController I0128 12:59:08.804861 1 container_runtime_config_controller.go:247] Starting MachineConfigController-ContainerRuntimeConfigController I0128 12:59:08.822442 1 reflector.go:376] Caches populated for *v1.PinnedImageSet from github.com/openshift/client-go/machineconfiguration/informers/externalversions/factory.go:125 I0128 12:59:08.867844 1 reflector.go:376] Caches populated for *v1.MachineConfig from github.com/openshift/client-go/machineconfiguration/informers/externalversions/factory.go:125 I0128 12:59:08.872720 1 reflector.go:376] Caches populated for *v1.MachineConfiguration from github.com/openshift/client-go/operator/informers/externalversions/factory.go:125 I0128 12:59:08.872819 1 machine_set_boot_image_controller.go:287] Bootimages management configuration has been added, reconciling enrolled machine resources I0128 12:59:08.891391 1 reflector.go:376] Caches populated for *v1.MachineOSConfig from github.com/openshift/client-go/machineconfiguration/informers/externalversions/factory.go:125 I0128 12:59:08.896579 1 pinned_image_set.go:115] Starting MachineConfigController-PinnedImageSetController I0128 12:59:08.905105 1 node_controller.go:250] Starting MachineConfigController-NodeController I0128 12:59:08.905233 1 template_controller.go:294] Starting MachineConfigController-TemplateController I0128 12:59:08.905436 1 machine_set_boot_image_controller.go:173] Starting MachineConfigController-MachineSetBootImageController I0128 12:59:08.905486 1 render_controller.go:141] Starting MachineConfigController-RenderController I0128 12:59:09.096004 1 kubelet_config_features.go:126] Applied FeatureSet cluster on MachineConfigPool master I0128 12:59:09.263437 1 kubelet_config_features.go:126] Applied FeatureSet cluster on MachineConfigPool worker I0128 12:59:10.046732 1 kubelet_config_nodes.go:163] Applied Node configuration 97-worker-generated-kubelet on MachineConfigPool worker I0128 12:59:10.611095 1 kubelet_config_nodes.go:163] Applied Node configuration 97-master-generated-kubelet on MachineConfigPool master I0128 12:59:11.211623 1 kubelet_config_features.go:126] Applied FeatureSet cluster on MachineConfigPool master I0128 12:59:11.811087 1 kubelet_config_features.go:126] Applied FeatureSet cluster on MachineConfigPool worker I0128 12:59:13.724329 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1d on AWS, with arch x86_64 I0128 12:59:13.725337 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1d I0128 12:59:13.743136 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1e on AWS, with arch x86_64 I0128 12:59:13.744083 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1e I0128 12:59:13.760902 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1f on AWS, with arch x86_64 I0128 12:59:13.761789 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1f I0128 12:59:14.115120 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1a on AWS, with arch x86_64 I0128 12:59:14.115883 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1a I0128 12:59:14.716727 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1b on AWS, with arch x86_64 I0128 12:59:14.717492 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1b I0128 12:59:15.315365 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1c on AWS, with arch x86_64 I0128 12:59:15.316171 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1c I0128 12:59:17.131292 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1e on AWS, with arch x86_64 I0128 12:59:17.132466 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1e I0128 12:59:17.715189 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1f on AWS, with arch x86_64 I0128 12:59:17.716818 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1f I0128 12:59:18.318461 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1a on AWS, with arch x86_64 I0128 12:59:18.319249 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1a I0128 12:59:18.917635 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1b on AWS, with arch x86_64 I0128 12:59:18.918792 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1b I0128 12:59:19.516792 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1c on AWS, with arch x86_64 I0128 12:59:19.517959 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1c I0128 12:59:20.116423 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1d on AWS, with arch x86_64 I0128 12:59:20.117200 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1d I0128 12:59:21.930432 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1a on AWS, with arch x86_64 I0128 12:59:21.931718 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1a I0128 12:59:22.534109 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1b on AWS, with arch x86_64 I0128 12:59:22.535277 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1b I0128 12:59:23.118161 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1c on AWS, with arch x86_64 I0128 12:59:23.118911 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1c I0128 12:59:23.724933 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1d on AWS, with arch x86_64 I0128 12:59:23.726065 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1d I0128 12:59:24.321935 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1e on AWS, with arch x86_64 I0128 12:59:24.323376 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1e I0128 12:59:24.917790 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1f on AWS, with arch x86_64 I0128 12:59:24.918953 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1f I0128 12:59:26.325055 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1e on AWS, with arch x86_64 I0128 12:59:26.326074 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1e I0128 12:59:26.921797 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1f on AWS, with arch x86_64 I0128 12:59:26.923187 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1f I0128 12:59:27.554813 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1a on AWS, with arch x86_64 I0128 12:59:27.557372 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1a I0128 12:59:28.119343 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1b on AWS, with arch x86_64 I0128 12:59:28.120529 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1b I0128 12:59:28.725485 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1c on AWS, with arch x86_64 I0128 12:59:28.727422 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1c I0128 12:59:29.839921 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1d on AWS, with arch x86_64 I0128 12:59:29.841200 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1d I0128 12:59:30.757600 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1a on AWS, with arch x86_64 I0128 12:59:30.759019 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1a I0128 12:59:31.335092 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1b on AWS, with arch x86_64 I0128 12:59:31.336325 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1b I0128 12:59:32.004962 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1c on AWS, with arch x86_64 I0128 12:59:32.006625 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1c I0128 12:59:32.532796 1 render_controller.go:560] Generated machineconfig rendered-master-cc0eb72f6670059591f052f019981c6a from 6 configs: [{MachineConfig 00-master machineconfiguration.openshift.io/v1 } {MachineConfig 01-master-container-runtime machineconfiguration.openshift.io/v1 } {MachineConfig 01-master-kubelet machineconfiguration.openshift.io/v1 } {MachineConfig 97-master-generated-kubelet machineconfiguration.openshift.io/v1 } {MachineConfig 98-master-generated-kubelet machineconfiguration.openshift.io/v1 } {MachineConfig 99-master-generated-registries machineconfiguration.openshift.io/v1 }] I0128 12:59:32.533233 1 event.go:377] Event(v1.ObjectReference{Kind:"MachineConfigPool", Namespace:"openshift-machine-config-operator", Name:"master", UID:"c25703a5-9e82-407f-ad5d-b4abb0e1439f", APIVersion:"machineconfiguration.openshift.io/v1", ResourceVersion:"37713", FieldPath:""}): type: 'Normal' reason: 'RenderedConfigGenerated' rendered-master-cc0eb72f6670059591f052f019981c6a successfully generated (release version: 4.19.21, controller version: 6fd31b7e48fefb01d263cd19b733aca57ba2cf95) I0128 12:59:32.534657 1 render_controller.go:560] Generated machineconfig rendered-worker-966255a81a98c722f0668af44f26a49d from 6 configs: [{MachineConfig 00-worker machineconfiguration.openshift.io/v1 } {MachineConfig 01-worker-container-runtime machineconfiguration.openshift.io/v1 } {MachineConfig 01-worker-kubelet machineconfiguration.openshift.io/v1 } {MachineConfig 97-worker-generated-kubelet machineconfiguration.openshift.io/v1 } {MachineConfig 98-worker-generated-kubelet machineconfiguration.openshift.io/v1 } {MachineConfig 99-worker-generated-registries machineconfiguration.openshift.io/v1 }] I0128 12:59:32.534834 1 event.go:377] Event(v1.ObjectReference{Kind:"MachineConfigPool", Namespace:"openshift-machine-config-operator", Name:"worker", UID:"465f13bc-fba0-47d6-bcdf-0a9946990ae3", APIVersion:"machineconfiguration.openshift.io/v1", ResourceVersion:"24550", FieldPath:""}): type: 'Normal' reason: 'RenderedConfigGenerated' rendered-worker-966255a81a98c722f0668af44f26a49d successfully generated (release version: 4.19.21, controller version: 6fd31b7e48fefb01d263cd19b733aca57ba2cf95) I0128 12:59:32.595746 1 render_controller.go:586] Pool worker: now targeting: rendered-worker-966255a81a98c722f0668af44f26a49d I0128 12:59:32.602303 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1d on AWS, with arch x86_64 I0128 12:59:32.603417 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1d I0128 12:59:32.673423 1 render_controller.go:586] Pool master: now targeting: rendered-master-cc0eb72f6670059591f052f019981c6a I0128 12:59:33.196046 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1e on AWS, with arch x86_64 I0128 12:59:33.197261 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1e I0128 12:59:33.716595 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1f on AWS, with arch x86_64 I0128 12:59:33.717791 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1f I0128 12:59:35.118295 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1d on AWS, with arch x86_64 I0128 12:59:35.119929 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1d I0128 12:59:35.766729 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1e on AWS, with arch x86_64 I0128 12:59:35.767854 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1e I0128 12:59:36.332017 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1f on AWS, with arch x86_64 I0128 12:59:36.333333 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1f I0128 12:59:36.918818 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1a on AWS, with arch x86_64 I0128 12:59:36.920091 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1a I0128 12:59:37.517403 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1b on AWS, with arch x86_64 I0128 12:59:37.518357 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1b I0128 12:59:37.644191 1 node_controller.go:607] Pool worker[zone=us-east-1a]: node ip-10-0-4-167.ec2.internal: changed taints I0128 12:59:37.679212 1 node_controller.go:607] Pool worker[zone=us-east-1b]: node ip-10-0-42-202.ec2.internal: changed taints I0128 12:59:37.713632 1 node_controller.go:1308] Pool worker: selected candidate node ip-10-0-4-167.ec2.internal I0128 12:59:37.713692 1 node_controller.go:1308] Pool worker: selected candidate node ip-10-0-42-202.ec2.internal I0128 12:59:37.713710 1 node_controller.go:1308] Pool worker: selected candidate node ip-10-0-83-25.ec2.internal I0128 12:59:37.713731 1 node_controller.go:597] Pool worker: 3 candidate nodes in 3 zones for update, capacity: 1 I0128 12:59:37.714096 1 node_controller.go:607] Pool worker[zone=us-east-1c]: node ip-10-0-83-25.ec2.internal: changed taints I0128 12:59:37.752004 1 node_controller.go:607] Pool master[zone=us-east-1b]: node ip-10-0-49-9.ec2.internal: changed taints I0128 12:59:37.822411 1 node_controller.go:607] Pool master[zone=us-east-1c]: node ip-10-0-66-141.ec2.internal: changed taints I0128 12:59:38.028877 1 node_controller.go:607] Pool worker[zone=us-east-1a]: node ip-10-0-4-167.ec2.internal: changed annotation machineconfiguration.openshift.io/desiredConfig = rendered-worker-966255a81a98c722f0668af44f26a49d I0128 12:59:38.030201 1 event.go:377] Event(v1.ObjectReference{Kind:"MachineConfigPool", Namespace:"openshift-machine-config-operator", Name:"worker", UID:"465f13bc-fba0-47d6-bcdf-0a9946990ae3", APIVersion:"machineconfiguration.openshift.io/v1", ResourceVersion:"38441", FieldPath:""}): type: 'Normal' reason: 'SetDesiredConfig' Targeted node ip-10-0-4-167.ec2.internal to MachineConfig: rendered-worker-966255a81a98c722f0668af44f26a49d I0128 12:59:38.134157 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1c on AWS, with arch x86_64 I0128 12:59:38.141640 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1c I0128 12:59:38.625097 1 node_controller.go:1308] Pool master: selected candidate node ip-10-0-49-9.ec2.internal I0128 12:59:38.625555 1 node_controller.go:1308] Pool master: selected candidate node ip-10-0-66-141.ec2.internal I0128 12:59:38.625629 1 node_controller.go:1308] Pool master: selected candidate node ip-10-0-21-80.ec2.internal I0128 12:59:38.625695 1 node_controller.go:597] Pool master: 3 candidate nodes in 3 zones for update, capacity: 1 I0128 12:59:38.625483 1 node_controller.go:607] Pool master[zone=us-east-1a]: node ip-10-0-21-80.ec2.internal: changed taints I0128 12:59:38.625996 1 node_controller.go:1357] Deferring update of machine config operator node: ip-10-0-66-141.ec2.internal I0128 12:59:38.626020 1 node_controller.go:597] Pool master: filtered to 2 candidate nodes for update, capacity: 1 I0128 12:59:38.626142 1 event.go:377] Event(v1.ObjectReference{Kind:"MachineConfigPool", Namespace:"openshift-machine-config-operator", Name:"master", UID:"c25703a5-9e82-407f-ad5d-b4abb0e1439f", APIVersion:"machineconfiguration.openshift.io/v1", ResourceVersion:"38445", FieldPath:""}): type: 'Normal' reason: 'DeferringOperatorNodeUpdate' Deferring update of machine config operator node ip-10-0-66-141.ec2.internal I0128 12:59:38.794577 1 node_controller.go:607] Pool worker[zone=us-east-1a]: node ip-10-0-4-167.ec2.internal: changed annotation machineconfiguration.openshift.io/state = Working I0128 12:59:39.239639 1 event.go:377] Event(v1.ObjectReference{Kind:"MachineConfigPool", Namespace:"openshift-machine-config-operator", Name:"master", UID:"c25703a5-9e82-407f-ad5d-b4abb0e1439f", APIVersion:"machineconfiguration.openshift.io/v1", ResourceVersion:"38445", FieldPath:""}): type: 'Normal' reason: 'SetDesiredConfig' Targeted node ip-10-0-21-80.ec2.internal to MachineConfig: rendered-master-cc0eb72f6670059591f052f019981c6a I0128 12:59:39.256321 1 node_controller.go:607] Pool master[zone=us-east-1a]: node ip-10-0-21-80.ec2.internal: changed annotation machineconfiguration.openshift.io/desiredConfig = rendered-master-cc0eb72f6670059591f052f019981c6a I0128 12:59:39.256434 1 event.go:377] Event(v1.ObjectReference{Kind:"MachineConfigPool", Namespace:"openshift-machine-config-operator", Name:"master", UID:"c25703a5-9e82-407f-ad5d-b4abb0e1439f", APIVersion:"machineconfiguration.openshift.io/v1", ResourceVersion:"38445", FieldPath:""}): type: 'Normal' reason: 'AnnotationChange' Node ip-10-0-21-80.ec2.internal now has machineconfiguration.openshift.io/desiredConfig=rendered-master-cc0eb72f6670059591f052f019981c6a I0128 12:59:39.525433 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1b on AWS, with arch x86_64 I0128 12:59:39.526715 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1b I0128 12:59:39.896198 1 node_controller.go:607] Pool master[zone=us-east-1a]: node ip-10-0-21-80.ec2.internal: changed annotation machineconfiguration.openshift.io/state = Working I0128 12:59:39.896448 1 event.go:377] Event(v1.ObjectReference{Kind:"MachineConfigPool", Namespace:"openshift-machine-config-operator", Name:"master", UID:"c25703a5-9e82-407f-ad5d-b4abb0e1439f", APIVersion:"machineconfiguration.openshift.io/v1", ResourceVersion:"39020", FieldPath:""}): type: 'Normal' reason: 'AnnotationChange' Node ip-10-0-21-80.ec2.internal now has machineconfiguration.openshift.io/state=Working I0128 12:59:40.121822 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1c on AWS, with arch x86_64 I0128 12:59:40.123037 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1c I0128 12:59:40.723539 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1d on AWS, with arch x86_64 I0128 12:59:40.725305 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1d I0128 12:59:41.317931 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1e on AWS, with arch x86_64 I0128 12:59:41.319163 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1e I0128 12:59:41.919690 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1f on AWS, with arch x86_64 I0128 12:59:41.920434 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1f I0128 12:59:42.527450 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1a on AWS, with arch x86_64 I0128 12:59:42.528602 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1a I0128 12:59:42.665598 1 node_controller.go:607] Pool worker[zone=us-east-1a]: node ip-10-0-4-167.ec2.internal: changed taints I0128 12:59:42.782642 1 node_controller.go:607] Pool master[zone=us-east-1a]: node ip-10-0-21-80.ec2.internal: changed taints I0128 12:59:43.923201 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1a on AWS, with arch x86_64 I0128 12:59:43.924133 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1a I0128 12:59:44.536451 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1b on AWS, with arch x86_64 I0128 12:59:44.537286 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1b I0128 12:59:45.123887 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1c on AWS, with arch x86_64 I0128 12:59:45.125094 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1c I0128 12:59:45.721224 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1d on AWS, with arch x86_64 I0128 12:59:45.723631 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1d I0128 12:59:46.320897 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1e on AWS, with arch x86_64 I0128 12:59:46.322092 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1e I0128 12:59:46.946969 1 platform_helpers.go:121] Reconciling MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1f on AWS, with arch x86_64 I0128 12:59:46.957186 1 machine_set_boot_image_controller.go:497] No patching required for MAPI machineset clusterpool419-z4ngz-swxvh-worker-us-east-1f I0128 12:59:53.482428 1 node_controller.go:607] Pool worker[zone=us-east-1a]: node ip-10-0-4-167.ec2.internal: Reporting unready: node ip-10-0-4-167.ec2.internal is reporting NotReady=False I0128 12:59:53.508887 1 node_controller.go:607] Pool worker[zone=us-east-1a]: node ip-10-0-4-167.ec2.internal: changed taints I0128 12:59:57.849954 1 node_controller.go:607] Pool worker[zone=us-east-1a]: node ip-10-0-4-167.ec2.internal: changed taints I0128 12:59:58.168655 1 drain_controller.go:193] node ip-10-0-4-167.ec2.internal: uncordoning I0128 12:59:58.168722 1 drain_controller.go:193] node ip-10-0-4-167.ec2.internal: initiating uncordon (currently schedulable: true) I0128 12:59:58.174411 1 drain_controller.go:193] node ip-10-0-4-167.ec2.internal: uncordon succeeded (currently schedulable: true) I0128 12:59:58.221632 1 drain_controller.go:193] node ip-10-0-4-167.ec2.internal: operation successful; applying completion annotation I0128 13:00:03.212603 1 node_controller.go:607] Pool worker[zone=us-east-1a]: node ip-10-0-4-167.ec2.internal: Completed update to rendered-worker-966255a81a98c722f0668af44f26a49d I0128 13:00:03.915929 1 node_controller.go:607] Pool worker[zone=us-east-1a]: node ip-10-0-4-167.ec2.internal: Reporting ready I0128 13:00:03.933791 1 node_controller.go:607] Pool worker[zone=us-east-1a]: node ip-10-0-4-167.ec2.internal: changed taints I0128 13:00:06.682027 1 drain_controller.go:193] node ip-10-0-21-80.ec2.internal: uncordoning I0128 13:00:06.682084 1 drain_controller.go:193] node ip-10-0-21-80.ec2.internal: initiating uncordon (currently schedulable: true) I0128 13:00:06.686371 1 drain_controller.go:193] node ip-10-0-21-80.ec2.internal: uncordon succeeded (currently schedulable: true) I0128 13:00:06.704576 1 drain_controller.go:193] node ip-10-0-21-80.ec2.internal: operation successful; applying completion annotation I0128 13:00:07.843730 1 node_controller.go:607] Pool worker[zone=us-east-1a]: node ip-10-0-4-167.ec2.internal: changed taints I0128 13:00:08.219970 1 node_controller.go:1308] Pool worker: selected candidate node ip-10-0-83-25.ec2.internal I0128 13:00:08.220009 1 node_controller.go:1308] Pool worker: selected candidate node ip-10-0-42-202.ec2.internal I0128 13:00:08.220019 1 node_controller.go:597] Pool worker: 2 candidate nodes in 2 zones for update, capacity: 1 I0128 13:00:08.248560 1 node_controller.go:607] Pool worker[zone=us-east-1b]: node ip-10-0-42-202.ec2.internal: changed annotation machineconfiguration.openshift.io/desiredConfig = rendered-worker-966255a81a98c722f0668af44f26a49d I0128 13:00:08.249613 1 event.go:377] Event(v1.ObjectReference{Kind:"MachineConfigPool", Namespace:"openshift-machine-config-operator", Name:"worker", UID:"465f13bc-fba0-47d6-bcdf-0a9946990ae3", APIVersion:"machineconfiguration.openshift.io/v1", ResourceVersion:"39099", FieldPath:""}): type: 'Normal' reason: 'SetDesiredConfig' Targeted node ip-10-0-42-202.ec2.internal to MachineConfig: rendered-worker-966255a81a98c722f0668af44f26a49d I0128 13:00:09.420200 1 node_controller.go:607] Pool worker[zone=us-east-1b]: node ip-10-0-42-202.ec2.internal: changed annotation machineconfiguration.openshift.io/state = Working I0128 13:00:11.706369 1 node_controller.go:607] Pool master[zone=us-east-1a]: node ip-10-0-21-80.ec2.internal: Completed update to rendered-master-cc0eb72f6670059591f052f019981c6a I0128 13:00:13.277295 1 node_controller.go:607] Pool worker[zone=us-east-1b]: node ip-10-0-42-202.ec2.internal: changed taints I0128 13:00:16.710919 1 node_controller.go:1308] Pool master: selected candidate node ip-10-0-49-9.ec2.internal I0128 13:00:16.710947 1 node_controller.go:1308] Pool master: selected candidate node ip-10-0-66-141.ec2.internal I0128 13:00:16.710955 1 node_controller.go:597] Pool master: 2 candidate nodes in 2 zones for update, capacity: 1 I0128 13:00:16.711084 1 node_controller.go:1357] Deferring update of machine config operator node: ip-10-0-66-141.ec2.internal I0128 13:00:16.711092 1 node_controller.go:597] Pool master: filtered to 1 candidate nodes for update, capacity: 1 I0128 13:00:16.711217 1 event.go:377] Event(v1.ObjectReference{Kind:"MachineConfigPool", Namespace:"openshift-machine-config-operator", Name:"master", UID:"c25703a5-9e82-407f-ad5d-b4abb0e1439f", APIVersion:"machineconfiguration.openshift.io/v1", ResourceVersion:"39110", FieldPath:""}): type: 'Normal' reason: 'DeferringOperatorNodeUpdate' Deferring update of machine config operator node ip-10-0-66-141.ec2.internal I0128 13:00:16.746138 1 node_controller.go:607] Pool master[zone=us-east-1b]: node ip-10-0-49-9.ec2.internal: changed annotation machineconfiguration.openshift.io/desiredConfig = rendered-master-cc0eb72f6670059591f052f019981c6a I0128 13:00:16.746528 1 event.go:377] Event(v1.ObjectReference{Kind:"MachineConfigPool", Namespace:"openshift-machine-config-operator", Name:"master", UID:"c25703a5-9e82-407f-ad5d-b4abb0e1439f", APIVersion:"machineconfiguration.openshift.io/v1", ResourceVersion:"39110", FieldPath:""}): type: 'Normal' reason: 'AnnotationChange' Node ip-10-0-49-9.ec2.internal now has machineconfiguration.openshift.io/desiredConfig=rendered-master-cc0eb72f6670059591f052f019981c6a I0128 13:00:16.748014 1 event.go:377] Event(v1.ObjectReference{Kind:"MachineConfigPool", Namespace:"openshift-machine-config-operator", Name:"master", UID:"c25703a5-9e82-407f-ad5d-b4abb0e1439f", APIVersion:"machineconfiguration.openshift.io/v1", ResourceVersion:"39110", FieldPath:""}): type: 'Normal' reason: 'SetDesiredConfig' Targeted node ip-10-0-49-9.ec2.internal to MachineConfig: rendered-master-cc0eb72f6670059591f052f019981c6a I0128 13:00:17.962545 1 node_controller.go:607] Pool master[zone=us-east-1b]: node ip-10-0-49-9.ec2.internal: changed annotation machineconfiguration.openshift.io/state = Working I0128 13:00:17.962898 1 event.go:377] Event(v1.ObjectReference{Kind:"MachineConfigPool", Namespace:"openshift-machine-config-operator", Name:"master", UID:"c25703a5-9e82-407f-ad5d-b4abb0e1439f", APIVersion:"machineconfiguration.openshift.io/v1", ResourceVersion:"40560", FieldPath:""}): type: 'Normal' reason: 'AnnotationChange' Node ip-10-0-49-9.ec2.internal now has machineconfiguration.openshift.io/state=Working I0128 13:00:19.263831 1 node_controller.go:607] Pool worker[zone=us-east-1b]: node ip-10-0-42-202.ec2.internal: Reporting unready: node ip-10-0-42-202.ec2.internal is reporting NotReady=False I0128 13:00:19.374936 1 node_controller.go:607] Pool worker[zone=us-east-1b]: node ip-10-0-42-202.ec2.internal: changed taints I0128 13:00:21.822269 1 node_controller.go:607] Pool master[zone=us-east-1b]: node ip-10-0-49-9.ec2.internal: changed taints I0128 13:00:22.995219 1 node_controller.go:607] Pool worker[zone=us-east-1b]: node ip-10-0-42-202.ec2.internal: changed taints I0128 13:00:23.887971 1 drain_controller.go:193] node ip-10-0-42-202.ec2.internal: uncordoning I0128 13:00:23.888127 1 drain_controller.go:193] node ip-10-0-42-202.ec2.internal: initiating uncordon (currently schedulable: true) I0128 13:00:23.897395 1 drain_controller.go:193] node ip-10-0-42-202.ec2.internal: uncordon succeeded (currently schedulable: true) I0128 13:00:23.925769 1 drain_controller.go:193] node ip-10-0-42-202.ec2.internal: operation successful; applying completion annotation I0128 13:00:28.919918 1 node_controller.go:607] Pool worker[zone=us-east-1b]: node ip-10-0-42-202.ec2.internal: Completed update to rendered-worker-966255a81a98c722f0668af44f26a49d I0128 13:00:29.659003 1 node_controller.go:607] Pool worker[zone=us-east-1b]: node ip-10-0-42-202.ec2.internal: Reporting ready I0128 13:00:29.690746 1 node_controller.go:607] Pool worker[zone=us-east-1b]: node ip-10-0-42-202.ec2.internal: changed taints I0128 13:00:32.943103 1 node_controller.go:607] Pool worker[zone=us-east-1b]: node ip-10-0-42-202.ec2.internal: changed taints I0128 13:00:34.119693 1 node_controller.go:1308] Pool worker: selected candidate node ip-10-0-83-25.ec2.internal I0128 13:00:34.119833 1 node_controller.go:597] Pool worker: 1 candidate nodes in 1 zones for update, capacity: 1 I0128 13:00:34.245498 1 node_controller.go:607] Pool worker[zone=us-east-1c]: node ip-10-0-83-25.ec2.internal: changed annotation machineconfiguration.openshift.io/desiredConfig = rendered-worker-966255a81a98c722f0668af44f26a49d I0128 13:00:34.245711 1 event.go:377] Event(v1.ObjectReference{Kind:"MachineConfigPool", Namespace:"openshift-machine-config-operator", Name:"worker", UID:"465f13bc-fba0-47d6-bcdf-0a9946990ae3", APIVersion:"machineconfiguration.openshift.io/v1", ResourceVersion:"40187", FieldPath:""}): type: 'Normal' reason: 'SetDesiredConfig' Targeted node ip-10-0-83-25.ec2.internal to MachineConfig: rendered-worker-966255a81a98c722f0668af44f26a49d I0128 13:00:35.397104 1 node_controller.go:607] Pool worker[zone=us-east-1c]: node ip-10-0-83-25.ec2.internal: changed annotation machineconfiguration.openshift.io/state = Working I0128 13:00:39.276004 1 node_controller.go:607] Pool worker[zone=us-east-1c]: node ip-10-0-83-25.ec2.internal: changed taints I0128 13:00:52.217671 1 drain_controller.go:193] node ip-10-0-83-25.ec2.internal: uncordoning I0128 13:00:52.217749 1 drain_controller.go:193] node ip-10-0-83-25.ec2.internal: initiating uncordon (currently schedulable: true) I0128 13:00:52.237131 1 drain_controller.go:193] node ip-10-0-83-25.ec2.internal: uncordon succeeded (currently schedulable: true) I0128 13:00:52.300119 1 drain_controller.go:193] node ip-10-0-83-25.ec2.internal: operation successful; applying completion annotation I0128 13:00:57.239561 1 node_controller.go:607] Pool worker[zone=us-east-1c]: node ip-10-0-83-25.ec2.internal: Completed update to rendered-worker-966255a81a98c722f0668af44f26a49d I0128 13:01:02.350346 1 status.go:266] Pool worker: All nodes are updated with MachineConfig rendered-worker-966255a81a98c722f0668af44f26a49d I0128 13:01:21.280721 1 drain_controller.go:193] node ip-10-0-49-9.ec2.internal: uncordoning I0128 13:01:21.280929 1 drain_controller.go:193] node ip-10-0-49-9.ec2.internal: initiating uncordon (currently schedulable: true) I0128 13:01:21.294748 1 drain_controller.go:193] node ip-10-0-49-9.ec2.internal: uncordon succeeded (currently schedulable: true) I0128 13:01:21.344138 1 drain_controller.go:193] node ip-10-0-49-9.ec2.internal: operation successful; applying completion annotation I0128 13:01:26.373929 1 node_controller.go:607] Pool master[zone=us-east-1b]: node ip-10-0-49-9.ec2.internal: Completed update to rendered-master-cc0eb72f6670059591f052f019981c6a I0128 13:01:31.389087 1 node_controller.go:1308] Pool master: selected candidate node ip-10-0-66-141.ec2.internal I0128 13:01:31.389205 1 node_controller.go:597] Pool master: 1 candidate nodes in 1 zones for update, capacity: 1 I0128 13:01:31.389225 1 node_controller.go:597] Pool master: filtered to 1 candidate nodes for update, capacity: 1 I0128 13:01:31.429183 1 event.go:377] Event(v1.ObjectReference{Kind:"MachineConfigPool", Namespace:"openshift-machine-config-operator", Name:"master", UID:"c25703a5-9e82-407f-ad5d-b4abb0e1439f", APIVersion:"machineconfiguration.openshift.io/v1", ResourceVersion:"40560", FieldPath:""}): type: 'Normal' reason: 'SetDesiredConfig' Targeted node ip-10-0-66-141.ec2.internal to MachineConfig: rendered-master-cc0eb72f6670059591f052f019981c6a I0128 13:01:31.434460 1 node_controller.go:607] Pool master[zone=us-east-1c]: node ip-10-0-66-141.ec2.internal: changed annotation machineconfiguration.openshift.io/desiredConfig = rendered-master-cc0eb72f6670059591f052f019981c6a I0128 13:01:31.434605 1 event.go:377] Event(v1.ObjectReference{Kind:"MachineConfigPool", Namespace:"openshift-machine-config-operator", Name:"master", UID:"c25703a5-9e82-407f-ad5d-b4abb0e1439f", APIVersion:"machineconfiguration.openshift.io/v1", ResourceVersion:"40560", FieldPath:""}): type: 'Normal' reason: 'AnnotationChange' Node ip-10-0-66-141.ec2.internal now has machineconfiguration.openshift.io/desiredConfig=rendered-master-cc0eb72f6670059591f052f019981c6a I0128 13:01:32.992361 1 node_controller.go:607] Pool master[zone=us-east-1c]: node ip-10-0-66-141.ec2.internal: changed annotation machineconfiguration.openshift.io/state = Working I0128 13:01:32.992808 1 event.go:377] Event(v1.ObjectReference{Kind:"MachineConfigPool", Namespace:"openshift-machine-config-operator", Name:"master", UID:"c25703a5-9e82-407f-ad5d-b4abb0e1439f", APIVersion:"machineconfiguration.openshift.io/v1", ResourceVersion:"46340", FieldPath:""}): type: 'Normal' reason: 'AnnotationChange' Node ip-10-0-66-141.ec2.internal now has machineconfiguration.openshift.io/state=Working I0128 13:01:36.538058 1 node_controller.go:607] Pool master[zone=us-east-1c]: node ip-10-0-66-141.ec2.internal: changed taints E0128 13:01:37.056872 1 render_controller.go:464] Error syncing Generated MCFG: Operation cannot be fulfilled on machineconfigpools.machineconfiguration.openshift.io "master": the object has been modified; please apply your changes to the latest version and try again E0128 13:01:37.074768 1 render_controller.go:486] Error updating MachineConfigPool master: Operation cannot be fulfilled on machineconfigpools.machineconfiguration.openshift.io "master": the object has been modified; please apply your changes to the latest version and try again I0128 13:01:37.074798 1 render_controller.go:394] Error syncing machineconfigpool master: Operation cannot be fulfilled on machineconfigpools.machineconfiguration.openshift.io "master": the object has been modified; please apply your changes to the latest version and try again I0128 13:02:07.511839 1 drain_controller.go:193] node ip-10-0-66-141.ec2.internal: uncordoning I0128 13:02:07.512058 1 drain_controller.go:193] node ip-10-0-66-141.ec2.internal: initiating uncordon (currently schedulable: true) I0128 13:02:07.543134 1 drain_controller.go:193] node ip-10-0-66-141.ec2.internal: uncordon succeeded (currently schedulable: true) I0128 13:02:07.639239 1 drain_controller.go:193] node ip-10-0-66-141.ec2.internal: operation successful; applying completion annotation I0128 13:02:12.568065 1 node_controller.go:607] Pool master[zone=us-east-1c]: node ip-10-0-66-141.ec2.internal: Completed update to rendered-master-cc0eb72f6670059591f052f019981c6a I0128 13:02:17.904997 1 status.go:266] Pool master: All nodes are updated with MachineConfig rendered-master-cc0eb72f6670059591f052f019981c6a I0128 13:20:21.068228 1 template_controller.go:146] Re-syncing ControllerConfig due to secret pull-secret change