Jul 27 21:03:27.761986 ip-10-0-128-201 systemd[1]: Starting Kubernetes Kubelet... Jul 27 21:03:28.128746 ip-10-0-128-201 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 27 21:03:28.128746 ip-10-0-128-201 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 27 21:03:28.128746 ip-10-0-128-201 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 27 21:03:28.128746 ip-10-0-128-201 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 27 21:03:28.128746 ip-10-0-128-201 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 27 21:03:28.131635 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.131516 2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 27 21:03:28.136640 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136616 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 21:03:28.136640 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136639 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 21:03:28.136640 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136644 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 21:03:28.136748 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136647 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 21:03:28.136748 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136651 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 21:03:28.136748 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136654 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 21:03:28.136748 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136657 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 21:03:28.136748 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136660 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 21:03:28.136748 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136663 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 21:03:28.136748 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136665 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 21:03:28.136748 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136668 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 21:03:28.136748 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136673 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 21:03:28.136748 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136677 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 21:03:28.136748 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136681 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 21:03:28.136748 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136685 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 21:03:28.136748 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136688 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 21:03:28.136748 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136691 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 21:03:28.136748 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136693 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 21:03:28.136748 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136696 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 21:03:28.136748 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136699 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 21:03:28.136748 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136703 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 21:03:28.136748 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136706 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 21:03:28.137216 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136709 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 21:03:28.137216 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136711 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 21:03:28.137216 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136714 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 21:03:28.137216 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136716 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 21:03:28.137216 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136719 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 21:03:28.137216 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136724 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 21:03:28.137216 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136728 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 21:03:28.137216 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136731 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 21:03:28.137216 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136733 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 21:03:28.137216 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136736 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 21:03:28.137216 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136739 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 21:03:28.137216 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136751 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 21:03:28.137216 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136754 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 21:03:28.137216 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136757 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 21:03:28.137216 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136760 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 21:03:28.137216 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136763 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 21:03:28.137216 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136765 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 21:03:28.137216 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136768 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 21:03:28.137216 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136770 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 21:03:28.137705 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136773 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 21:03:28.137705 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136775 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 21:03:28.137705 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136778 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 21:03:28.137705 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136781 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 21:03:28.137705 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136783 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 21:03:28.137705 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136786 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 21:03:28.137705 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136789 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 21:03:28.137705 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136792 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 21:03:28.137705 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136794 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 21:03:28.137705 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136796 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 21:03:28.137705 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136799 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 21:03:28.137705 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136801 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 21:03:28.137705 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136804 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 21:03:28.137705 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136807 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 27 21:03:28.137705 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136810 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 21:03:28.137705 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136813 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 21:03:28.137705 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136816 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 21:03:28.137705 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136818 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 21:03:28.137705 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136820 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 21:03:28.137705 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136823 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 21:03:28.138212 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136825 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 21:03:28.138212 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136828 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 21:03:28.138212 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136830 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 21:03:28.138212 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136833 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 21:03:28.138212 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136836 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 21:03:28.138212 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136838 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 21:03:28.138212 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136841 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 21:03:28.138212 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136843 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 21:03:28.138212 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136846 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 21:03:28.138212 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136848 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 21:03:28.138212 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136852 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 21:03:28.138212 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136854 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 21:03:28.138212 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136857 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 21:03:28.138212 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136860 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 21:03:28.138212 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136862 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 21:03:28.138212 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136866 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 21:03:28.138212 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136869 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 21:03:28.138212 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136871 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 21:03:28.138212 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136873 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 21:03:28.138212 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136876 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 21:03:28.138761 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136878 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 21:03:28.138761 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136881 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 21:03:28.138761 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136883 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 21:03:28.138761 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136886 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 21:03:28.138761 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136888 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 21:03:28.138761 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136891 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 21:03:28.138761 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136895 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 21:03:28.138761 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136897 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 21:03:28.138761 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136900 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 21:03:28.138761 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136903 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 21:03:28.138761 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136905 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 21:03:28.138761 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136908 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 21:03:28.138761 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136910 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 21:03:28.138761 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136913 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 21:03:28.138761 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136915 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 21:03:28.138761 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136918 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 21:03:28.138761 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136921 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 21:03:28.138761 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.136924 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 21:03:28.138761 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137714 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 21:03:28.138761 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137723 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 21:03:28.139279 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137727 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 21:03:28.139279 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137731 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 21:03:28.139279 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137736 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 21:03:28.139279 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137740 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 21:03:28.139279 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137743 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 21:03:28.139279 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137746 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 27 21:03:28.139279 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137749 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 21:03:28.139279 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137752 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 21:03:28.139279 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137754 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 21:03:28.139279 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137757 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 21:03:28.139279 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137760 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 21:03:28.139279 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137762 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 21:03:28.139279 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137765 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 21:03:28.139279 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137767 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 21:03:28.139279 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137770 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 21:03:28.139279 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137772 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 21:03:28.139279 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137775 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 21:03:28.139279 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137779 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 21:03:28.139279 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137782 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 21:03:28.139279 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137785 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 21:03:28.139808 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137787 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 21:03:28.139808 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137790 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 21:03:28.139808 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137793 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 21:03:28.139808 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137796 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 21:03:28.139808 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137798 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 21:03:28.139808 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137801 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 21:03:28.139808 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137803 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 21:03:28.139808 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137806 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 21:03:28.139808 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137808 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 21:03:28.139808 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137810 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 21:03:28.139808 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137813 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 21:03:28.139808 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137815 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 21:03:28.139808 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137817 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 21:03:28.139808 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137820 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 21:03:28.139808 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137822 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 21:03:28.139808 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137825 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 21:03:28.139808 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137829 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 21:03:28.139808 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137831 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 21:03:28.139808 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137834 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 21:03:28.139808 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137836 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 21:03:28.140327 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137840 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 21:03:28.140327 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137843 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 21:03:28.140327 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137846 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 21:03:28.140327 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137849 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 21:03:28.140327 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137852 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 21:03:28.140327 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137855 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 21:03:28.140327 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137858 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 21:03:28.140327 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137861 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 21:03:28.140327 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137863 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 21:03:28.140327 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137866 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 21:03:28.140327 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137870 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 21:03:28.140327 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137872 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 21:03:28.140327 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137876 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 21:03:28.140327 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137878 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 21:03:28.140327 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137881 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 21:03:28.140327 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137884 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 21:03:28.140327 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137886 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 21:03:28.140327 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137889 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 21:03:28.140327 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137891 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 21:03:28.140852 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137894 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 21:03:28.140852 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137896 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 21:03:28.140852 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137899 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 21:03:28.140852 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137901 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 21:03:28.140852 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137904 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 21:03:28.140852 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137906 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 21:03:28.140852 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137910 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 21:03:28.140852 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137913 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 21:03:28.140852 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137915 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 21:03:28.140852 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137918 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 21:03:28.140852 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137921 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 21:03:28.140852 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137923 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 21:03:28.140852 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137925 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 21:03:28.140852 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137928 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 21:03:28.140852 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137930 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 21:03:28.140852 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137933 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 21:03:28.140852 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137935 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 21:03:28.140852 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137938 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 21:03:28.140852 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137941 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 21:03:28.140852 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137943 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 21:03:28.141351 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137946 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 21:03:28.141351 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137948 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 21:03:28.141351 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137951 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 21:03:28.141351 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137954 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 21:03:28.141351 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137956 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 21:03:28.141351 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137959 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 21:03:28.141351 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137962 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 21:03:28.141351 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137964 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 21:03:28.141351 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137967 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 21:03:28.141351 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.137969 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 21:03:28.141351 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138005 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 21:03:28.141351 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138034 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 21:03:28.141351 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138041 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 21:03:28.141351 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138044 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 21:03:28.141351 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138048 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 21:03:28.141351 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138052 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 21:03:28.141351 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138055 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 21:03:28.141351 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138058 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 21:03:28.141351 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138495 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 21:03:28.141847 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138501 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 21:03:28.141847 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138505 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 21:03:28.141847 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138508 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 21:03:28.141847 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138511 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 21:03:28.141847 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138514 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 21:03:28.141847 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138517 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 21:03:28.141847 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138520 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 21:03:28.141847 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138522 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 21:03:28.141847 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138525 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 21:03:28.141847 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138528 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 21:03:28.141847 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138530 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 21:03:28.141847 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138533 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 21:03:28.141847 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138536 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 21:03:28.141847 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138538 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 21:03:28.141847 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138541 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 21:03:28.141847 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138543 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 21:03:28.141847 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138546 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 21:03:28.141847 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138548 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 21:03:28.141847 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138551 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 21:03:28.141847 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138554 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 21:03:28.142439 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138557 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 21:03:28.142439 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138560 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 21:03:28.142439 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138562 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 21:03:28.142439 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138565 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 21:03:28.142439 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138586 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 21:03:28.142439 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138590 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 21:03:28.142439 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138594 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 21:03:28.142439 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138597 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 21:03:28.142439 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138600 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 21:03:28.142439 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138602 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 21:03:28.142439 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138605 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 21:03:28.142439 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138607 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 21:03:28.142439 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138610 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 21:03:28.142439 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138612 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 21:03:28.142439 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138615 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 21:03:28.142439 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138618 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 21:03:28.142439 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138621 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 21:03:28.142439 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138623 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 21:03:28.142439 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138626 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 21:03:28.142439 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138628 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 21:03:28.142939 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138631 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 27 21:03:28.142939 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138634 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 21:03:28.142939 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138637 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 21:03:28.142939 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138639 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 21:03:28.142939 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138642 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 21:03:28.142939 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138644 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 21:03:28.142939 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138647 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 21:03:28.142939 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138649 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 21:03:28.142939 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138651 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 21:03:28.142939 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138655 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 21:03:28.142939 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138659 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 21:03:28.142939 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138663 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 21:03:28.142939 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138666 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 21:03:28.142939 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138668 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 21:03:28.142939 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138671 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 21:03:28.142939 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138674 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 21:03:28.142939 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138678 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 21:03:28.142939 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138682 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 21:03:28.142939 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138685 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 21:03:28.143440 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138689 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 21:03:28.143440 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138692 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 21:03:28.143440 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138695 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 21:03:28.143440 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138698 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 21:03:28.143440 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138701 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 21:03:28.143440 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138704 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 21:03:28.143440 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138707 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 21:03:28.143440 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138711 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 21:03:28.143440 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138714 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 21:03:28.143440 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138716 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 21:03:28.143440 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138719 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 21:03:28.143440 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138722 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 21:03:28.143440 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138724 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 21:03:28.143440 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138727 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 21:03:28.143440 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138730 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 21:03:28.143440 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138732 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 21:03:28.143440 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138735 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 21:03:28.143440 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138737 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 21:03:28.143440 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138740 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 21:03:28.143440 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138748 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 21:03:28.143986 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138750 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 21:03:28.143986 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138753 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 21:03:28.143986 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138755 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 21:03:28.143986 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138758 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 21:03:28.143986 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138761 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 21:03:28.143986 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138763 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 21:03:28.143986 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138766 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 21:03:28.143986 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138769 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 21:03:28.143986 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138771 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 21:03:28.143986 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138775 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 21:03:28.143986 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138783 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 21:03:28.143986 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138786 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 21:03:28.143986 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138789 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 21:03:28.143986 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138791 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 21:03:28.143986 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138794 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 21:03:28.143986 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138796 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 21:03:28.143986 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138799 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 21:03:28.143986 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138802 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 21:03:28.143986 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.138804 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 21:03:28.143986 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.138883 2577 flags.go:64] FLAG: --address="0.0.0.0" Jul 27 21:03:28.144492 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.138891 2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 27 21:03:28.144492 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.138899 2577 flags.go:64] FLAG: --anonymous-auth="true" Jul 27 21:03:28.144492 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.138904 2577 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 27 21:03:28.144492 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.138908 2577 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 27 21:03:28.144492 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.138911 2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 27 21:03:28.144492 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.138915 2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 27 21:03:28.144492 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.138925 2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 27 21:03:28.144492 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.138928 2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 27 21:03:28.144492 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.138931 2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 27 21:03:28.144492 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.138935 2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 27 21:03:28.144492 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.138938 2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 27 21:03:28.144492 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.138942 2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 27 21:03:28.144492 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.138945 2577 flags.go:64] FLAG: --cgroup-root="" Jul 27 21:03:28.144492 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.138949 2577 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 27 21:03:28.144492 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.138952 2577 flags.go:64] FLAG: --client-ca-file="" Jul 27 21:03:28.144492 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.138955 2577 flags.go:64] FLAG: --cloud-provider="external" Jul 27 21:03:28.144492 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.138959 2577 flags.go:64] FLAG: --cluster-dns="[]" Jul 27 21:03:28.144492 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.138963 2577 flags.go:64] FLAG: --cluster-domain="" Jul 27 21:03:28.144492 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.138966 2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 27 21:03:28.144492 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.138969 2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 27 21:03:28.144492 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.138973 2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 27 21:03:28.144492 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.138976 2577 flags.go:64] FLAG: --container-log-max-files="5" Jul 27 21:03:28.144492 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.138980 2577 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 27 21:03:28.144492 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.138984 2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 27 21:03:28.145100 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.138987 2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 27 21:03:28.145100 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.138990 2577 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 27 21:03:28.145100 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.138993 2577 flags.go:64] FLAG: --contention-profiling="false" Jul 27 21:03:28.145100 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.138996 2577 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 27 21:03:28.145100 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.138999 2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 27 21:03:28.145100 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139003 2577 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 27 21:03:28.145100 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139005 2577 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 27 21:03:28.145100 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139010 2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 27 21:03:28.145100 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139013 2577 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 27 21:03:28.145100 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139016 2577 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 27 21:03:28.145100 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139019 2577 flags.go:64] FLAG: --enable-load-reader="false" Jul 27 21:03:28.145100 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139022 2577 flags.go:64] FLAG: --enable-server="true" Jul 27 21:03:28.145100 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139025 2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 27 21:03:28.145100 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139030 2577 flags.go:64] FLAG: --event-burst="100" Jul 27 21:03:28.145100 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139033 2577 flags.go:64] FLAG: --event-qps="50" Jul 27 21:03:28.145100 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139036 2577 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 27 21:03:28.145100 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139039 2577 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 27 21:03:28.145100 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139042 2577 flags.go:64] FLAG: --eviction-hard="" Jul 27 21:03:28.145100 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139045 2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 27 21:03:28.145100 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139049 2577 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 27 21:03:28.145100 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139052 2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 27 21:03:28.145100 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139055 2577 flags.go:64] FLAG: --eviction-soft="" Jul 27 21:03:28.145100 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139058 2577 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 27 21:03:28.145100 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139061 2577 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 27 21:03:28.145100 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139064 2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 27 21:03:28.145829 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139067 2577 flags.go:64] FLAG: --experimental-mounter-path="" Jul 27 21:03:28.145829 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139070 2577 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 27 21:03:28.145829 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139073 2577 flags.go:64] FLAG: --fail-swap-on="true" Jul 27 21:03:28.145829 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139076 2577 flags.go:64] FLAG: --feature-gates="" Jul 27 21:03:28.145829 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139080 2577 flags.go:64] FLAG: --file-check-frequency="20s" Jul 27 21:03:28.145829 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139083 2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 27 21:03:28.145829 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139086 2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 27 21:03:28.145829 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139089 2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 27 21:03:28.145829 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139092 2577 flags.go:64] FLAG: --healthz-port="10248" Jul 27 21:03:28.145829 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139095 2577 flags.go:64] FLAG: --help="false" Jul 27 21:03:28.145829 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139099 2577 flags.go:64] FLAG: --hostname-override="ip-10-0-128-201.ec2.internal" Jul 27 21:03:28.145829 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139102 2577 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 27 21:03:28.145829 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139105 2577 flags.go:64] FLAG: --http-check-frequency="20s" Jul 27 21:03:28.145829 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139108 2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 27 21:03:28.145829 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139112 2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 27 21:03:28.145829 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139115 2577 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 27 21:03:28.145829 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139118 2577 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 27 21:03:28.145829 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139121 2577 flags.go:64] FLAG: --image-service-endpoint="" Jul 27 21:03:28.145829 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139124 2577 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 27 21:03:28.145829 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139127 2577 flags.go:64] FLAG: --kube-api-burst="100" Jul 27 21:03:28.145829 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139130 2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 27 21:03:28.145829 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139133 2577 flags.go:64] FLAG: --kube-api-qps="50" Jul 27 21:03:28.145829 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139136 2577 flags.go:64] FLAG: --kube-reserved="" Jul 27 21:03:28.145829 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139139 2577 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 27 21:03:28.146476 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139142 2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 27 21:03:28.146476 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139145 2577 flags.go:64] FLAG: --kubelet-cgroups="" Jul 27 21:03:28.146476 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139148 2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 27 21:03:28.146476 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139151 2577 flags.go:64] FLAG: --lock-file="" Jul 27 21:03:28.146476 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139154 2577 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 27 21:03:28.146476 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139157 2577 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 27 21:03:28.146476 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139160 2577 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 27 21:03:28.146476 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139166 2577 flags.go:64] FLAG: --log-json-split-stream="false" Jul 27 21:03:28.146476 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139169 2577 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 27 21:03:28.146476 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139171 2577 flags.go:64] FLAG: --log-text-split-stream="false" Jul 27 21:03:28.146476 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139174 2577 flags.go:64] FLAG: --logging-format="text" Jul 27 21:03:28.146476 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139177 2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 27 21:03:28.146476 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139180 2577 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 27 21:03:28.146476 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139184 2577 flags.go:64] FLAG: --manifest-url="" Jul 27 21:03:28.146476 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139186 2577 flags.go:64] FLAG: --manifest-url-header="" Jul 27 21:03:28.146476 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139191 2577 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 27 21:03:28.146476 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139195 2577 flags.go:64] FLAG: --max-open-files="1000000" Jul 27 21:03:28.146476 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139199 2577 flags.go:64] FLAG: --max-pods="110" Jul 27 21:03:28.146476 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139202 2577 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 27 21:03:28.146476 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139205 2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 27 21:03:28.146476 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139208 2577 flags.go:64] FLAG: --memory-manager-policy="None" Jul 27 21:03:28.146476 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139211 2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 27 21:03:28.146476 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139214 2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 27 21:03:28.146476 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139216 2577 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 27 21:03:28.146476 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139220 2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 27 21:03:28.147147 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139228 2577 flags.go:64] FLAG: --node-status-max-images="50" Jul 27 21:03:28.147147 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139231 2577 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 27 21:03:28.147147 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139233 2577 flags.go:64] FLAG: --oom-score-adj="-999" Jul 27 21:03:28.147147 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139236 2577 flags.go:64] FLAG: --pod-cidr="" Jul 27 21:03:28.147147 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139239 2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 27 21:03:28.147147 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139245 2577 flags.go:64] FLAG: --pod-manifest-path="" Jul 27 21:03:28.147147 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139248 2577 flags.go:64] FLAG: --pod-max-pids="-1" Jul 27 21:03:28.147147 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139251 2577 flags.go:64] FLAG: --pods-per-core="0" Jul 27 21:03:28.147147 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139254 2577 flags.go:64] FLAG: --port="10250" Jul 27 21:03:28.147147 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139257 2577 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 27 21:03:28.147147 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139260 2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f97584116477a527" Jul 27 21:03:28.147147 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139263 2577 flags.go:64] FLAG: --qos-reserved="" Jul 27 21:03:28.147147 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139266 2577 flags.go:64] FLAG: --read-only-port="10255" Jul 27 21:03:28.147147 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139269 2577 flags.go:64] FLAG: --register-node="true" Jul 27 21:03:28.147147 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139272 2577 flags.go:64] FLAG: --register-with-taints="" Jul 27 21:03:28.147147 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139276 2577 flags.go:64] FLAG: --registry-burst="10" Jul 27 21:03:28.147147 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139279 2577 flags.go:64] FLAG: --registry-qps="5" Jul 27 21:03:28.147147 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139282 2577 flags.go:64] FLAG: --reserved-cpus="" Jul 27 21:03:28.147147 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139285 2577 flags.go:64] FLAG: --reserved-memory="" Jul 27 21:03:28.147147 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139289 2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 27 21:03:28.147147 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139292 2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 27 21:03:28.147147 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139296 2577 flags.go:64] FLAG: --rotate-certificates="false" Jul 27 21:03:28.147147 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139299 2577 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 27 21:03:28.147147 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139301 2577 flags.go:64] FLAG: --runonce="false" Jul 27 21:03:28.147147 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139305 2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 27 21:03:28.147820 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139308 2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 27 21:03:28.147820 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139311 2577 flags.go:64] FLAG: --seccomp-default="false" Jul 27 21:03:28.147820 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139314 2577 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 27 21:03:28.147820 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139316 2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 27 21:03:28.147820 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139319 2577 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 27 21:03:28.147820 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139323 2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 27 21:03:28.147820 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139326 2577 flags.go:64] FLAG: --storage-driver-password="root" Jul 27 21:03:28.147820 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139329 2577 flags.go:64] FLAG: --storage-driver-secure="false" Jul 27 21:03:28.147820 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139332 2577 flags.go:64] FLAG: --storage-driver-table="stats" Jul 27 21:03:28.147820 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139337 2577 flags.go:64] FLAG: --storage-driver-user="root" Jul 27 21:03:28.147820 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139340 2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 27 21:03:28.147820 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139358 2577 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 27 21:03:28.147820 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139362 2577 flags.go:64] FLAG: --system-cgroups="" Jul 27 21:03:28.147820 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139365 2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 27 21:03:28.147820 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139372 2577 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 27 21:03:28.147820 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139375 2577 flags.go:64] FLAG: --tls-cert-file="" Jul 27 21:03:28.147820 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139378 2577 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 27 21:03:28.147820 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139382 2577 flags.go:64] FLAG: --tls-min-version="" Jul 27 21:03:28.147820 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139386 2577 flags.go:64] FLAG: --tls-private-key-file="" Jul 27 21:03:28.147820 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139389 2577 flags.go:64] FLAG: --topology-manager-policy="none" Jul 27 21:03:28.147820 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139392 2577 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 27 21:03:28.147820 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139395 2577 flags.go:64] FLAG: --topology-manager-scope="container" Jul 27 21:03:28.147820 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139398 2577 flags.go:64] FLAG: --v="2" Jul 27 21:03:28.147820 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139402 2577 flags.go:64] FLAG: --version="false" Jul 27 21:03:28.147820 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139406 2577 flags.go:64] FLAG: --vmodule="" Jul 27 21:03:28.148452 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139410 2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 27 21:03:28.148452 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.139413 2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 27 21:03:28.148452 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139542 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 21:03:28.148452 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139545 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 21:03:28.148452 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139549 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 21:03:28.148452 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139551 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 21:03:28.148452 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139554 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 21:03:28.148452 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139557 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 21:03:28.148452 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139560 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 21:03:28.148452 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139563 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 21:03:28.148452 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139566 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 21:03:28.148452 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139582 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 21:03:28.148452 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139586 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 21:03:28.148452 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139591 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 21:03:28.148452 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139595 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 21:03:28.148452 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139599 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 21:03:28.148452 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139604 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 21:03:28.148452 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139607 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 21:03:28.148452 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139610 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 21:03:28.148452 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139613 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 21:03:28.148971 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139615 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 21:03:28.148971 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139619 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 21:03:28.148971 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139623 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 21:03:28.148971 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139626 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 21:03:28.148971 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139629 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 21:03:28.148971 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139634 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 21:03:28.148971 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139637 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 21:03:28.148971 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139640 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 27 21:03:28.148971 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139643 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 21:03:28.148971 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139646 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 21:03:28.148971 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139649 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 21:03:28.148971 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139652 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 21:03:28.148971 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139655 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 21:03:28.148971 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139657 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 21:03:28.148971 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139660 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 21:03:28.148971 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139663 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 21:03:28.148971 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139666 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 21:03:28.148971 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139668 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 21:03:28.148971 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139671 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 21:03:28.148971 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139673 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 21:03:28.149487 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139677 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 21:03:28.149487 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139679 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 21:03:28.149487 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139682 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 21:03:28.149487 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139685 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 21:03:28.149487 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139688 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 21:03:28.149487 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139690 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 21:03:28.149487 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139693 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 21:03:28.149487 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139695 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 21:03:28.149487 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139699 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 21:03:28.149487 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139701 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 21:03:28.149487 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139704 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 21:03:28.149487 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139706 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 21:03:28.149487 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139709 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 21:03:28.149487 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139712 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 21:03:28.149487 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139714 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 21:03:28.149487 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139717 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 21:03:28.149487 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139720 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 21:03:28.149487 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139724 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 21:03:28.149487 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139726 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 21:03:28.149487 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139729 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 21:03:28.150048 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139732 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 21:03:28.150048 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139734 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 21:03:28.150048 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139737 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 21:03:28.150048 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139739 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 21:03:28.150048 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139742 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 21:03:28.150048 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139744 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 21:03:28.150048 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139747 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 21:03:28.150048 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139749 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 21:03:28.150048 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139753 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 21:03:28.150048 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139755 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 21:03:28.150048 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139758 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 21:03:28.150048 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139760 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 21:03:28.150048 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139763 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 21:03:28.150048 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139766 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 21:03:28.150048 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139768 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 21:03:28.150048 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139771 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 21:03:28.150048 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139773 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 21:03:28.150048 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139776 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 21:03:28.150048 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139778 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 21:03:28.150048 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139781 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 21:03:28.150549 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139784 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 21:03:28.150549 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139787 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 21:03:28.150549 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139789 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 21:03:28.150549 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139792 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 21:03:28.150549 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139795 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 21:03:28.150549 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139797 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 21:03:28.150549 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139800 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 21:03:28.150549 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139802 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 21:03:28.150549 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139805 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 21:03:28.150549 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139809 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 21:03:28.150549 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139811 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 21:03:28.150549 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139814 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 21:03:28.150549 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139817 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 21:03:28.150549 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139819 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 21:03:28.150549 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139822 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 21:03:28.150549 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139825 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 21:03:28.150549 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139827 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 21:03:28.150549 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139831 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 21:03:28.150549 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139835 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 21:03:28.151042 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139837 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 21:03:28.151042 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.139841 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 21:03:28.151042 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.140687 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 27 21:03:28.151042 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.148030 2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 27 21:03:28.151042 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.148050 2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 27 21:03:28.151042 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.148072 2577 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 27 21:03:28.151042 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.148079 2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 27 21:03:28.151042 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148144 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 21:03:28.151042 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148150 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 21:03:28.151042 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148153 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 21:03:28.151042 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148157 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 21:03:28.151042 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148160 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 21:03:28.151042 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148163 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 21:03:28.151042 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148168 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 21:03:28.151042 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148173 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 21:03:28.151416 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148176 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 21:03:28.151416 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148179 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 21:03:28.151416 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148181 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 21:03:28.151416 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148184 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 21:03:28.151416 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148187 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 21:03:28.151416 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148190 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 21:03:28.151416 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148193 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 21:03:28.151416 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148195 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 21:03:28.151416 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148198 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 21:03:28.151416 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148200 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 21:03:28.151416 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148204 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 21:03:28.151416 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148206 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 21:03:28.151416 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148209 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 21:03:28.151416 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148212 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 21:03:28.151416 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148214 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 21:03:28.151416 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148217 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 21:03:28.151416 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148219 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 21:03:28.151416 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148222 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 21:03:28.151416 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148225 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 21:03:28.151416 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148228 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 21:03:28.151975 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148230 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 21:03:28.151975 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148234 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 21:03:28.151975 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148237 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 21:03:28.151975 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148240 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 21:03:28.151975 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148243 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 21:03:28.151975 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148246 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 21:03:28.151975 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148248 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 21:03:28.151975 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148251 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 21:03:28.151975 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148254 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 21:03:28.151975 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148256 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 21:03:28.151975 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148259 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 21:03:28.151975 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148261 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 21:03:28.151975 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148264 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 21:03:28.151975 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148267 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 21:03:28.151975 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148269 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 21:03:28.151975 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148272 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 21:03:28.151975 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148275 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 21:03:28.151975 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148277 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 21:03:28.151975 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148279 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 21:03:28.151975 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148282 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 21:03:28.152496 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148284 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 21:03:28.152496 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148287 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 21:03:28.152496 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148289 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 21:03:28.152496 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148292 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 21:03:28.152496 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148294 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 21:03:28.152496 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148297 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 21:03:28.152496 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148300 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 21:03:28.152496 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148302 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 21:03:28.152496 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148305 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 21:03:28.152496 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148307 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 21:03:28.152496 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148310 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 21:03:28.152496 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148312 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 21:03:28.152496 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148315 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 21:03:28.152496 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148317 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 21:03:28.152496 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148320 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 21:03:28.152496 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148323 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 21:03:28.152496 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148325 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 21:03:28.152496 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148328 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 21:03:28.152496 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148331 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 21:03:28.152995 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148334 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 21:03:28.152995 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148337 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 21:03:28.152995 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148339 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 21:03:28.152995 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148342 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 21:03:28.152995 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148344 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 21:03:28.152995 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148347 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 21:03:28.152995 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148350 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 21:03:28.152995 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148353 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 21:03:28.152995 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148355 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 21:03:28.152995 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148357 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 21:03:28.152995 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148360 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 21:03:28.152995 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148363 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 21:03:28.152995 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148367 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 21:03:28.152995 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148371 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 21:03:28.152995 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148373 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 21:03:28.152995 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148376 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 21:03:28.152995 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148379 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 27 21:03:28.152995 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148383 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 21:03:28.152995 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148387 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 21:03:28.153507 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148390 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 21:03:28.153507 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148393 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 21:03:28.153507 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148396 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 21:03:28.153507 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148399 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 21:03:28.153507 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148402 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 21:03:28.153507 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148405 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 21:03:28.153507 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148407 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 21:03:28.153507 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148410 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 21:03:28.153507 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148413 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 21:03:28.153507 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148415 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 21:03:28.153507 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148418 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 21:03:28.153507 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148420 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 21:03:28.153507 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148423 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 21:03:28.153507 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.148429 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 27 21:03:28.153507 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148549 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 21:03:28.154003 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148553 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 21:03:28.154003 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148556 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 21:03:28.154003 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148558 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 21:03:28.154003 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148561 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 21:03:28.154003 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148564 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 21:03:28.154003 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148584 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 21:03:28.154003 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148589 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 21:03:28.154003 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148592 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 21:03:28.154003 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148596 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 21:03:28.154003 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148599 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 21:03:28.154003 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148602 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 21:03:28.154003 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148605 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 21:03:28.154003 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148607 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 21:03:28.154003 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148610 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 21:03:28.154003 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148614 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 21:03:28.154003 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148617 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 21:03:28.154003 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148620 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 21:03:28.154003 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148622 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 21:03:28.154003 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148625 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 21:03:28.154003 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148628 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 21:03:28.154492 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148630 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 21:03:28.154492 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148633 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 21:03:28.154492 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148636 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 21:03:28.154492 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148638 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 21:03:28.154492 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148641 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 21:03:28.154492 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148643 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 21:03:28.154492 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148646 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 21:03:28.154492 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148649 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 21:03:28.154492 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148652 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 21:03:28.154492 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148654 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 21:03:28.154492 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148658 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 21:03:28.154492 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148660 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 21:03:28.154492 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148663 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 21:03:28.154492 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148667 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 21:03:28.154492 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148671 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 21:03:28.154492 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148674 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 21:03:28.154492 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148677 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 21:03:28.154492 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148679 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 21:03:28.154492 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148682 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 21:03:28.154492 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148684 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 21:03:28.154992 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148686 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 21:03:28.154992 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148689 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 21:03:28.154992 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148691 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 21:03:28.154992 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148694 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 21:03:28.154992 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148696 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 21:03:28.154992 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148699 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 21:03:28.154992 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148702 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 21:03:28.154992 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148705 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 21:03:28.154992 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148707 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 21:03:28.154992 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148710 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 21:03:28.154992 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148712 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 21:03:28.154992 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148715 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 21:03:28.154992 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148718 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 21:03:28.154992 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148722 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 21:03:28.154992 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148725 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 21:03:28.154992 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148728 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 21:03:28.154992 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148731 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 21:03:28.154992 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148734 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 21:03:28.154992 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148737 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 21:03:28.155447 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148740 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 21:03:28.155447 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148743 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 21:03:28.155447 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148746 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 21:03:28.155447 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148749 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 21:03:28.155447 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148752 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 21:03:28.155447 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148754 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 21:03:28.155447 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148757 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 21:03:28.155447 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148759 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 21:03:28.155447 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148762 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 21:03:28.155447 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148764 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 21:03:28.155447 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148767 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 21:03:28.155447 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148769 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 21:03:28.155447 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148772 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 21:03:28.155447 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148775 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 21:03:28.155447 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148777 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 21:03:28.155447 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148779 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 21:03:28.155447 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148782 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 21:03:28.155447 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148784 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 21:03:28.155447 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148787 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 21:03:28.155447 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148790 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 27 21:03:28.155970 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148792 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 21:03:28.155970 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148795 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 21:03:28.155970 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148798 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 21:03:28.155970 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148800 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 21:03:28.155970 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148803 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 21:03:28.155970 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148805 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 21:03:28.155970 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148808 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 21:03:28.155970 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148810 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 21:03:28.155970 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148812 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 21:03:28.155970 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148815 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 21:03:28.155970 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148819 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 21:03:28.155970 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148822 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 21:03:28.155970 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148826 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 21:03:28.155970 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148829 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 21:03:28.155970 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148831 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 21:03:28.155970 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148834 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 21:03:28.155970 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148836 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 21:03:28.155970 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148839 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 21:03:28.155970 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:28.148842 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 21:03:28.156698 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.148847 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 27 21:03:28.156698 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.148981 2577 server.go:962] "Client rotation is on, will bootstrap in background" Jul 27 21:03:28.156698 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.151502 2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 27 21:03:28.156698 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.153259 2577 server.go:1019] "Starting client certificate rotation" Jul 27 21:03:28.156698 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.153362 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 27 21:03:28.156698 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.154253 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 27 21:03:28.156698 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.156336 2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 27 21:03:28.162520 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.162495 2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 27 21:03:28.174261 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.174236 2577 log.go:25] "Validated CRI v1 runtime API" Jul 27 21:03:28.176937 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.176915 2577 log.go:25] "Validated CRI v1 image API" Jul 27 21:03:28.178710 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.178683 2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 27 21:03:28.181365 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.181330 2577 fs.go:135] Filesystem UUIDs: map[26cef3fb-5c09-441f-82ac-39dbdc222070:/dev/nvme0n1p3 4ef4ac14-cd6b-41fc-a838-4460a6268237:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Jul 27 21:03:28.181471 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.181363 2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 27 21:03:28.183124 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.183108 2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 27 21:03:28.186726 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.186595 2577 manager.go:216] Machine: {Timestamp:2026-07-27 21:03:28.185226002 +0000 UTC m=+0.316842217 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3094067 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29427a7a238fa78d8ae89d669b276a SystemUUID:ec29427a-7a23-8fa7-8d8a-e89d669b276a BootID:911b2e13-6110-4f99-ad1b-fd73121319d0 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:63:28:76:c3:95 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:63:28:76:c3:95 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c2:e2:cd:32:22:70 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 27 21:03:28.186726 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.186718 2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 27 21:03:28.186853 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.186817 2577 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 27 21:03:28.188073 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.188038 2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 27 21:03:28.188219 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.188075 2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-201.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 27 21:03:28.189833 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.189819 2577 topology_manager.go:138] "Creating topology manager with none policy" Jul 27 21:03:28.189833 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.189834 2577 container_manager_linux.go:309] "Creating device plugin manager" Jul 27 21:03:28.189908 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.189843 2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 27 21:03:28.190658 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.190646 2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 27 21:03:28.191916 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.191903 2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 27 21:03:28.193791 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.193775 2577 state_mem.go:36] "Initialized new in-memory state store" Jul 27 21:03:28.193919 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.193910 2577 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 27 21:03:28.193951 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.193944 2577 kubelet.go:486] "Attempting to sync node with API server" Jul 27 21:03:28.193982 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.193956 2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 27 21:03:28.193982 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.193975 2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 27 21:03:28.194036 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.193986 2577 kubelet.go:398] "Adding apiserver pod source" Jul 27 21:03:28.194036 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.194005 2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 27 21:03:28.194401 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.194389 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 27 21:03:28.194449 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.194408 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 27 21:03:28.197812 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.197790 2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 27 21:03:28.199759 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.199744 2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 27 21:03:28.199818 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.199776 2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 27 21:03:28.201870 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.201854 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 27 21:03:28.201870 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.201872 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 27 21:03:28.201984 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.201882 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 27 21:03:28.201984 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.201889 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 27 21:03:28.201984 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.201898 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 27 21:03:28.201984 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.201904 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 27 21:03:28.201984 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.201913 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 27 21:03:28.201984 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.201918 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 27 21:03:28.201984 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.201926 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 27 21:03:28.201984 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.201932 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 27 21:03:28.201984 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.201940 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 27 21:03:28.201984 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.201946 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 27 21:03:28.203769 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.203740 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 27 21:03:28.204771 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.204752 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 27 21:03:28.207438 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.207409 2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-201.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 27 21:03:28.208059 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:28.208020 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-201.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 27 21:03:28.208315 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:28.208012 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 27 21:03:28.208596 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.208565 2577 server.go:1268] "Started kubelet" Jul 27 21:03:28.208684 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.208646 2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 27 21:03:28.208796 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.208741 2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 27 21:03:28.208900 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.208823 2577 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 27 21:03:28.209553 ip-10-0-128-201 systemd[1]: Started Kubernetes Kubelet. Jul 27 21:03:28.210831 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.210811 2577 server.go:310] "Adding debug handlers to kubelet server" Jul 27 21:03:28.210932 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.210896 2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 27 21:03:28.213958 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:28.213939 2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 27 21:03:28.217145 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:28.214937 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-201.ec2.internal.18c64165e6323031 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-201.ec2.internal,UID:ip-10-0-128-201.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-201.ec2.internal,},FirstTimestamp:2026-07-27 21:03:28.208531505 +0000 UTC m=+0.340147721,LastTimestamp:2026-07-27 21:03:28.208531505 +0000 UTC m=+0.340147721,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-201.ec2.internal,}" Jul 27 21:03:28.220988 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.220967 2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 27 21:03:28.220988 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.220974 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 27 21:03:28.221151 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:28.221076 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-201.ec2.internal\" not found" Jul 27 21:03:28.221838 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.221794 2577 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 27 21:03:28.221838 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.221792 2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 27 21:03:28.221838 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.221812 2577 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 27 21:03:28.222023 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.221881 2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 27 21:03:28.222023 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.221895 2577 factory.go:55] Registering systemd factory Jul 27 21:03:28.222023 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.221905 2577 factory.go:223] Registration of the systemd container factory successfully Jul 27 21:03:28.222023 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.221948 2577 reconstruct.go:97] "Volume reconstruction finished" Jul 27 21:03:28.222023 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.221960 2577 reconciler.go:29] "Reconciler: start to sync state" Jul 27 21:03:28.222243 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.222137 2577 factory.go:153] Registering CRI-O factory Jul 27 21:03:28.222243 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.222148 2577 factory.go:223] Registration of the crio container factory successfully Jul 27 21:03:28.222243 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.222169 2577 factory.go:103] Registering Raw factory Jul 27 21:03:28.222243 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.222179 2577 manager.go:1196] Started watching for new ooms in manager Jul 27 21:03:28.222561 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.222549 2577 manager.go:349] Starting recovery of all containers Jul 27 21:03:28.223088 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:28.223060 2577 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-201.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 27 21:03:28.223182 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:28.223100 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 27 21:03:28.232292 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.232252 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-khswn" Jul 27 21:03:28.233595 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.233563 2577 manager.go:354] Recovery completed Jul 27 21:03:28.240037 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.240015 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-khswn" Jul 27 21:03:28.240266 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.240251 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 21:03:28.242266 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.242242 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-201.ec2.internal" event="NodeHasSufficientMemory" Jul 27 21:03:28.242335 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.242282 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-201.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 21:03:28.242335 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.242294 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-201.ec2.internal" event="NodeHasSufficientPID" Jul 27 21:03:28.242930 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.242908 2577 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 27 21:03:28.242930 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.242927 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 21:03:28.243061 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.242928 2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 27 21:03:28.243061 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.243027 2577 state_mem.go:36] "Initialized new in-memory state store" Jul 27 21:03:28.243619 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.243605 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-201.ec2.internal" event="NodeHasSufficientMemory" Jul 27 21:03:28.243667 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.243635 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-201.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 21:03:28.243667 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.243650 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-201.ec2.internal" event="NodeHasSufficientPID" Jul 27 21:03:28.244614 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.244600 2577 policy_none.go:49] "None policy: Start" Jul 27 21:03:28.244685 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.244621 2577 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 27 21:03:28.244685 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.244637 2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 27 21:03:28.245635 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.245622 2577 policy_none.go:47] "Start" Jul 27 21:03:28.274243 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.274224 2577 manager.go:337] "Starting Device Plugin manager" Jul 27 21:03:28.274351 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:28.274336 2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 27 21:03:28.274410 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.274352 2577 server.go:87] "Starting device plugin registration server" Jul 27 21:03:28.274622 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.274607 2577 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 27 21:03:28.274731 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.274623 2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 27 21:03:28.274731 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.274717 2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 27 21:03:28.274859 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.274804 2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 27 21:03:28.274859 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.274814 2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 27 21:03:28.275288 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:28.275266 2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 27 21:03:28.275364 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:28.275324 2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-201.ec2.internal\" not found" Jul 27 21:03:28.297396 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.297364 2577 container_manager_linux.go:676] "Failed to ensure state" containerName="/system.slice" err="failed to move PID 2610 into the system container \"/system.slice\": " Jul 27 21:03:28.358615 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.358565 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 27 21:03:28.360057 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.360037 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 27 21:03:28.360160 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.360079 2577 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 27 21:03:28.360160 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.360102 2577 kubelet.go:2448] "Starting kubelet main sync loop" Jul 27 21:03:28.360160 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:28.360138 2577 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 27 21:03:28.362475 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.362452 2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 21:03:28.375469 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.375437 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 21:03:28.376549 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.376524 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-201.ec2.internal" event="NodeHasSufficientMemory" Jul 27 21:03:28.376684 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.376561 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-201.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 21:03:28.376684 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.376590 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-201.ec2.internal" event="NodeHasSufficientPID" Jul 27 21:03:28.376684 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.376620 2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-201.ec2.internal" Jul 27 21:03:28.385701 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.385606 2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-201.ec2.internal" Jul 27 21:03:28.385701 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:28.385639 2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-201.ec2.internal\": node \"ip-10-0-128-201.ec2.internal\" not found" Jul 27 21:03:28.412359 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:28.412187 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-201.ec2.internal\" not found" Jul 27 21:03:28.460886 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.460806 2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-201.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-201.ec2.internal"] Jul 27 21:03:28.461023 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.460933 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 21:03:28.462623 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.462604 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-201.ec2.internal" event="NodeHasSufficientMemory" Jul 27 21:03:28.462733 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.462635 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-201.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 21:03:28.462733 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.462649 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-201.ec2.internal" event="NodeHasSufficientPID" Jul 27 21:03:28.463883 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.463870 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 21:03:28.464040 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.464026 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-201.ec2.internal" Jul 27 21:03:28.464078 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.464057 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 21:03:28.464615 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.464598 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-201.ec2.internal" event="NodeHasSufficientMemory" Jul 27 21:03:28.464715 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.464621 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-201.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 21:03:28.464715 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.464630 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-201.ec2.internal" event="NodeHasSufficientPID" Jul 27 21:03:28.464715 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.464598 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-201.ec2.internal" event="NodeHasSufficientMemory" Jul 27 21:03:28.464715 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.464698 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-201.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 21:03:28.464715 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.464710 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-201.ec2.internal" event="NodeHasSufficientPID" Jul 27 21:03:28.465736 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.465721 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-201.ec2.internal" Jul 27 21:03:28.465802 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.465752 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 21:03:28.466393 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.466376 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-201.ec2.internal" event="NodeHasSufficientMemory" Jul 27 21:03:28.466472 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.466433 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-201.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 21:03:28.466472 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.466447 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-201.ec2.internal" event="NodeHasSufficientPID" Jul 27 21:03:28.494468 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:28.494442 2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-201.ec2.internal\" not found" node="ip-10-0-128-201.ec2.internal" Jul 27 21:03:28.498842 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:28.498821 2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-201.ec2.internal\" not found" node="ip-10-0-128-201.ec2.internal" Jul 27 21:03:28.512859 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:28.512832 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-201.ec2.internal\" not found" Jul 27 21:03:28.523982 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.523957 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bbc03a6c330f6e41568ceafa19359664-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-201.ec2.internal\" (UID: \"bbc03a6c330f6e41568ceafa19359664\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-201.ec2.internal" Jul 27 21:03:28.524049 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.524001 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bbc03a6c330f6e41568ceafa19359664-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-201.ec2.internal\" (UID: \"bbc03a6c330f6e41568ceafa19359664\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-201.ec2.internal" Jul 27 21:03:28.524049 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.524036 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4572a1a28313da7538ca6395c7dccdb4-config\") pod \"kube-apiserver-proxy-ip-10-0-128-201.ec2.internal\" (UID: \"4572a1a28313da7538ca6395c7dccdb4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-201.ec2.internal" Jul 27 21:03:28.613564 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:28.613520 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-201.ec2.internal\" not found" Jul 27 21:03:28.624993 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.624963 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bbc03a6c330f6e41568ceafa19359664-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-201.ec2.internal\" (UID: \"bbc03a6c330f6e41568ceafa19359664\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-201.ec2.internal" Jul 27 21:03:28.625046 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.625004 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4572a1a28313da7538ca6395c7dccdb4-config\") pod \"kube-apiserver-proxy-ip-10-0-128-201.ec2.internal\" (UID: \"4572a1a28313da7538ca6395c7dccdb4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-201.ec2.internal" Jul 27 21:03:28.625046 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.625022 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bbc03a6c330f6e41568ceafa19359664-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-201.ec2.internal\" (UID: \"bbc03a6c330f6e41568ceafa19359664\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-201.ec2.internal" Jul 27 21:03:28.625111 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.625074 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bbc03a6c330f6e41568ceafa19359664-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-201.ec2.internal\" (UID: \"bbc03a6c330f6e41568ceafa19359664\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-201.ec2.internal" Jul 27 21:03:28.625111 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.625090 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bbc03a6c330f6e41568ceafa19359664-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-201.ec2.internal\" (UID: \"bbc03a6c330f6e41568ceafa19359664\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-201.ec2.internal" Jul 27 21:03:28.625170 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.625091 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4572a1a28313da7538ca6395c7dccdb4-config\") pod \"kube-apiserver-proxy-ip-10-0-128-201.ec2.internal\" (UID: \"4572a1a28313da7538ca6395c7dccdb4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-201.ec2.internal" Jul 27 21:03:28.714494 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:28.714413 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-201.ec2.internal\" not found" Jul 27 21:03:28.797022 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.796987 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-201.ec2.internal" Jul 27 21:03:28.801812 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:28.801786 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-201.ec2.internal" Jul 27 21:03:28.815073 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:28.815042 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-201.ec2.internal\" not found" Jul 27 21:03:28.915622 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:28.915562 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-201.ec2.internal\" not found" Jul 27 21:03:29.016212 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:29.016112 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-201.ec2.internal\" not found" Jul 27 21:03:29.117329 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:29.117277 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-201.ec2.internal\" not found" Jul 27 21:03:29.153328 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:29.153295 2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 27 21:03:29.153832 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:29.153470 2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 27 21:03:29.218259 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:29.218226 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-201.ec2.internal\" not found" Jul 27 21:03:29.245934 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:29.245883 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-26 20:58:28 +0000 UTC" deadline="2028-05-11 12:45:51.490417494 +0000 UTC" Jul 27 21:03:29.245934 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:29.245922 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15687h42m22.244499259s" Jul 27 21:03:29.318701 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:29.318658 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-201.ec2.internal\" not found" Jul 27 21:03:29.419639 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:29.419602 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-201.ec2.internal\" not found" Jul 27 21:03:29.421632 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:29.421597 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbc03a6c330f6e41568ceafa19359664.slice/crio-5d37e7842f73ed1ea3899e779988c4a203569b89619936fc6c61a341f41b091e WatchSource:0}: Error finding container 5d37e7842f73ed1ea3899e779988c4a203569b89619936fc6c61a341f41b091e: Status 404 returned error can't find the container with id 5d37e7842f73ed1ea3899e779988c4a203569b89619936fc6c61a341f41b091e Jul 27 21:03:29.422126 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:29.422105 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4572a1a28313da7538ca6395c7dccdb4.slice/crio-13ced1d046807abcadc5d5b77700e1f9534044958eb9373285fa4b62c163bf9f WatchSource:0}: Error finding container 13ced1d046807abcadc5d5b77700e1f9534044958eb9373285fa4b62c163bf9f: Status 404 returned error can't find the container with id 13ced1d046807abcadc5d5b77700e1f9534044958eb9373285fa4b62c163bf9f Jul 27 21:03:29.426890 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:29.426864 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 27 21:03:29.496396 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:29.496349 2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 21:03:29.507022 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:29.506997 2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 21:03:29.521906 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:29.521878 2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-201.ec2.internal" Jul 27 21:03:29.536919 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:29.536894 2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 27 21:03:29.537934 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:29.537918 2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-201.ec2.internal" Jul 27 21:03:29.547400 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:29.547375 2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 27 21:03:29.589670 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:29.589597 2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 21:03:30.196669 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.196630 2577 apiserver.go:52] "Watching apiserver" Jul 27 21:03:30.203646 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.203455 2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 27 21:03:30.203953 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.203922 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-npnjm","openshift-multus/multus-tv2n5","openshift-multus/network-metrics-daemon-lvg8z","openshift-network-diagnostics/network-check-target-lrnv5","openshift-network-operator/iptables-alerter-64nks","openshift-ovn-kubernetes/ovnkube-node-l6mhn","kube-system/global-pull-secret-syncer-l2czx","kube-system/konnectivity-agent-tx4cs","openshift-cluster-node-tuning-operator/tuned-hq8kb","openshift-image-registry/node-ca-zt59k","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-201.ec2.internal","openshift-multus/multus-additional-cni-plugins-5kfcg","kube-system/kube-apiserver-proxy-ip-10-0-128-201.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jz7ws"] Jul 27 21:03:30.205588 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.205559 2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 21:03:30.205871 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.205851 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2czx" Jul 27 21:03:30.205986 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:30.205959 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2czx" podUID="c77555ba-462e-49c5-b8b0-e83987e8a2e3" Jul 27 21:03:30.207177 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.207141 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tx4cs" Jul 27 21:03:30.208727 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.208558 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lvg8z" Jul 27 21:03:30.208727 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:30.208676 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lvg8z" podUID="0c32154a-b41d-45c4-afa6-240a8ea24835" Jul 27 21:03:30.211645 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.211422 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 27 21:03:30.211645 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.211510 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-9jplt\"" Jul 27 21:03:30.211841 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.211755 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 27 21:03:30.212922 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.212883 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lrnv5" Jul 27 21:03:30.213039 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:30.212961 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lrnv5" podUID="8157bc07-d223-44de-aee6-ffa0c683a2cb" Jul 27 21:03:30.214949 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.214927 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-64nks" Jul 27 21:03:30.216354 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.216334 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" Jul 27 21:03:30.217451 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.217427 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 27 21:03:30.217534 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.217492 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 27 21:03:30.217798 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.217779 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 27 21:03:30.218087 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.218064 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-mvtfq\"" Jul 27 21:03:30.218338 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.218319 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hq8kb" Jul 27 21:03:30.219037 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.218975 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 27 21:03:30.219110 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.219055 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 27 21:03:30.219773 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.219750 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-npnjm" Jul 27 21:03:30.221827 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.221601 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tv2n5" Jul 27 21:03:30.222018 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.221835 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 27 21:03:30.222018 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.222007 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 27 21:03:30.222697 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.222668 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 27 21:03:30.222795 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.222723 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 27 21:03:30.222908 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.222893 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 27 21:03:30.223810 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.223784 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-b4dmt\"" Jul 27 21:03:30.224227 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.224208 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-djnzn\"" Jul 27 21:03:30.224297 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.224252 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 27 21:03:30.225469 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.225444 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 27 21:03:30.227895 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.225934 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 27 21:03:30.227895 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.226556 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 27 21:03:30.227895 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.226718 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 27 21:03:30.227895 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.227360 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-gspzp\"" Jul 27 21:03:30.227895 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.227506 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-t9p8x\"" Jul 27 21:03:30.227895 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.227634 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 27 21:03:30.227895 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.227786 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 27 21:03:30.229755 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.229530 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 27 21:03:30.232891 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.232429 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zt59k" Jul 27 21:03:30.234728 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.234136 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8addb952-4638-47ca-a4fb-d04fb3441aa9-cnibin\") pod \"multus-tv2n5\" (UID: \"8addb952-4638-47ca-a4fb-d04fb3441aa9\") " pod="openshift-multus/multus-tv2n5" Jul 27 21:03:30.234728 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.234164 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1907dc14-1132-4eb9-a5ee-675b43df6e6d-host-slash\") pod \"ovnkube-node-l6mhn\" (UID: \"1907dc14-1132-4eb9-a5ee-675b43df6e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" Jul 27 21:03:30.234728 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.234181 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1907dc14-1132-4eb9-a5ee-675b43df6e6d-etc-openvswitch\") pod \"ovnkube-node-l6mhn\" (UID: \"1907dc14-1132-4eb9-a5ee-675b43df6e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" Jul 27 21:03:30.234728 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.234196 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6f4b2b00-fd6c-463e-b27a-e1a925557c2f-lib-modules\") pod \"tuned-hq8kb\" (UID: \"6f4b2b00-fd6c-463e-b27a-e1a925557c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-hq8kb" Jul 27 21:03:30.234728 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.234214 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8addb952-4638-47ca-a4fb-d04fb3441aa9-cni-binary-copy\") pod \"multus-tv2n5\" (UID: \"8addb952-4638-47ca-a4fb-d04fb3441aa9\") " pod="openshift-multus/multus-tv2n5" Jul 27 21:03:30.234728 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.234220 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jz7ws" Jul 27 21:03:30.234728 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.234259 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p2lc\" (UniqueName: \"kubernetes.io/projected/0c32154a-b41d-45c4-afa6-240a8ea24835-kube-api-access-7p2lc\") pod \"network-metrics-daemon-lvg8z\" (UID: \"0c32154a-b41d-45c4-afa6-240a8ea24835\") " pod="openshift-multus/network-metrics-daemon-lvg8z" Jul 27 21:03:30.234728 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.234318 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1907dc14-1132-4eb9-a5ee-675b43df6e6d-run-systemd\") pod \"ovnkube-node-l6mhn\" (UID: \"1907dc14-1132-4eb9-a5ee-675b43df6e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" Jul 27 21:03:30.234728 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.234351 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1907dc14-1132-4eb9-a5ee-675b43df6e6d-ovnkube-config\") pod \"ovnkube-node-l6mhn\" (UID: \"1907dc14-1132-4eb9-a5ee-675b43df6e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" Jul 27 21:03:30.234728 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.234376 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6f4b2b00-fd6c-463e-b27a-e1a925557c2f-tmp\") pod \"tuned-hq8kb\" (UID: \"6f4b2b00-fd6c-463e-b27a-e1a925557c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-hq8kb" Jul 27 21:03:30.234728 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.234420 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq4kc\" (UniqueName: \"kubernetes.io/projected/8addb952-4638-47ca-a4fb-d04fb3441aa9-kube-api-access-zq4kc\") pod \"multus-tv2n5\" (UID: \"8addb952-4638-47ca-a4fb-d04fb3441aa9\") " pod="openshift-multus/multus-tv2n5" Jul 27 21:03:30.234728 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.234448 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1907dc14-1132-4eb9-a5ee-675b43df6e6d-host-kubelet\") pod \"ovnkube-node-l6mhn\" (UID: \"1907dc14-1132-4eb9-a5ee-675b43df6e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" Jul 27 21:03:30.234728 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.234492 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8addb952-4638-47ca-a4fb-d04fb3441aa9-host-run-netns\") pod \"multus-tv2n5\" (UID: \"8addb952-4638-47ca-a4fb-d04fb3441aa9\") " pod="openshift-multus/multus-tv2n5" Jul 27 21:03:30.234728 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.234519 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c2ebad9-de50-4346-a64f-6cceecf44434-tmp-dir\") pod \"node-resolver-npnjm\" (UID: \"4c2ebad9-de50-4346-a64f-6cceecf44434\") " pod="openshift-dns/node-resolver-npnjm" Jul 27 21:03:30.234728 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.234572 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4w4r\" (UniqueName: \"kubernetes.io/projected/4c2ebad9-de50-4346-a64f-6cceecf44434-kube-api-access-s4w4r\") pod \"node-resolver-npnjm\" (UID: \"4c2ebad9-de50-4346-a64f-6cceecf44434\") " pod="openshift-dns/node-resolver-npnjm" Jul 27 21:03:30.234728 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.234598 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjdsl\" (UniqueName: \"kubernetes.io/projected/1907dc14-1132-4eb9-a5ee-675b43df6e6d-kube-api-access-bjdsl\") pod \"ovnkube-node-l6mhn\" (UID: \"1907dc14-1132-4eb9-a5ee-675b43df6e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" Jul 27 21:03:30.234728 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.234706 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6f4b2b00-fd6c-463e-b27a-e1a925557c2f-etc-sysctl-conf\") pod \"tuned-hq8kb\" (UID: \"6f4b2b00-fd6c-463e-b27a-e1a925557c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-hq8kb" Jul 27 21:03:30.236217 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.234740 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8addb952-4638-47ca-a4fb-d04fb3441aa9-multus-cni-dir\") pod \"multus-tv2n5\" (UID: \"8addb952-4638-47ca-a4fb-d04fb3441aa9\") " pod="openshift-multus/multus-tv2n5" Jul 27 21:03:30.236217 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.234781 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e7bf9687-782f-40c9-bb73-e8b4b2c6a763-serviceca\") pod \"node-ca-zt59k\" (UID: \"e7bf9687-782f-40c9-bb73-e8b4b2c6a763\") " pod="openshift-image-registry/node-ca-zt59k" Jul 27 21:03:30.236217 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.234825 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c77555ba-462e-49c5-b8b0-e83987e8a2e3-original-pull-secret\") pod \"global-pull-secret-syncer-l2czx\" (UID: \"c77555ba-462e-49c5-b8b0-e83987e8a2e3\") " pod="kube-system/global-pull-secret-syncer-l2czx" Jul 27 21:03:30.236217 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.234855 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6f4b2b00-fd6c-463e-b27a-e1a925557c2f-etc-tuned\") pod \"tuned-hq8kb\" (UID: \"6f4b2b00-fd6c-463e-b27a-e1a925557c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-hq8kb" Jul 27 21:03:30.236217 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.234898 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8addb952-4638-47ca-a4fb-d04fb3441aa9-host-run-multus-certs\") pod \"multus-tv2n5\" (UID: \"8addb952-4638-47ca-a4fb-d04fb3441aa9\") " pod="openshift-multus/multus-tv2n5" Jul 27 21:03:30.236217 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.234930 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e7bf9687-782f-40c9-bb73-e8b4b2c6a763-host\") pod \"node-ca-zt59k\" (UID: \"e7bf9687-782f-40c9-bb73-e8b4b2c6a763\") " pod="openshift-image-registry/node-ca-zt59k" Jul 27 21:03:30.236217 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.234964 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c32154a-b41d-45c4-afa6-240a8ea24835-metrics-certs\") pod \"network-metrics-daemon-lvg8z\" (UID: \"0c32154a-b41d-45c4-afa6-240a8ea24835\") " pod="openshift-multus/network-metrics-daemon-lvg8z" Jul 27 21:03:30.236217 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.234993 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft79h\" (UniqueName: \"kubernetes.io/projected/8157bc07-d223-44de-aee6-ffa0c683a2cb-kube-api-access-ft79h\") pod \"network-check-target-lrnv5\" (UID: \"8157bc07-d223-44de-aee6-ffa0c683a2cb\") " pod="openshift-network-diagnostics/network-check-target-lrnv5" Jul 27 21:03:30.236217 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.235025 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c77555ba-462e-49c5-b8b0-e83987e8a2e3-dbus\") pod \"global-pull-secret-syncer-l2czx\" (UID: \"c77555ba-462e-49c5-b8b0-e83987e8a2e3\") " pod="kube-system/global-pull-secret-syncer-l2czx" Jul 27 21:03:30.236217 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.235052 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f8ef5264-95a3-4193-ae7d-f0f10a7733ab-host-slash\") pod \"iptables-alerter-64nks\" (UID: \"f8ef5264-95a3-4193-ae7d-f0f10a7733ab\") " pod="openshift-network-operator/iptables-alerter-64nks" Jul 27 21:03:30.236217 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.235077 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fsh5\" (UniqueName: \"kubernetes.io/projected/6f4b2b00-fd6c-463e-b27a-e1a925557c2f-kube-api-access-8fsh5\") pod \"tuned-hq8kb\" (UID: \"6f4b2b00-fd6c-463e-b27a-e1a925557c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-hq8kb" Jul 27 21:03:30.236217 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.235105 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8addb952-4638-47ca-a4fb-d04fb3441aa9-os-release\") pod \"multus-tv2n5\" (UID: \"8addb952-4638-47ca-a4fb-d04fb3441aa9\") " pod="openshift-multus/multus-tv2n5" Jul 27 21:03:30.236217 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.235169 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f8ef5264-95a3-4193-ae7d-f0f10a7733ab-iptables-alerter-script\") pod \"iptables-alerter-64nks\" (UID: \"f8ef5264-95a3-4193-ae7d-f0f10a7733ab\") " pod="openshift-network-operator/iptables-alerter-64nks" Jul 27 21:03:30.236217 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.235197 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6f4b2b00-fd6c-463e-b27a-e1a925557c2f-etc-modprobe-d\") pod \"tuned-hq8kb\" (UID: \"6f4b2b00-fd6c-463e-b27a-e1a925557c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-hq8kb" Jul 27 21:03:30.236217 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.235231 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/de32d9f8-5f7d-4a68-8aa4-2830c11db8f5-agent-certs\") pod \"konnectivity-agent-tx4cs\" (UID: \"de32d9f8-5f7d-4a68-8aa4-2830c11db8f5\") " pod="kube-system/konnectivity-agent-tx4cs" Jul 27 21:03:30.236217 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.235268 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8addb952-4638-47ca-a4fb-d04fb3441aa9-multus-socket-dir-parent\") pod \"multus-tv2n5\" (UID: \"8addb952-4638-47ca-a4fb-d04fb3441aa9\") " pod="openshift-multus/multus-tv2n5" Jul 27 21:03:30.237049 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.235296 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8addb952-4638-47ca-a4fb-d04fb3441aa9-host-var-lib-cni-bin\") pod \"multus-tv2n5\" (UID: \"8addb952-4638-47ca-a4fb-d04fb3441aa9\") " pod="openshift-multus/multus-tv2n5" Jul 27 21:03:30.237049 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.235323 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6f4b2b00-fd6c-463e-b27a-e1a925557c2f-etc-systemd\") pod \"tuned-hq8kb\" (UID: \"6f4b2b00-fd6c-463e-b27a-e1a925557c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-hq8kb" Jul 27 21:03:30.237049 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.235349 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4c2ebad9-de50-4346-a64f-6cceecf44434-hosts-file\") pod \"node-resolver-npnjm\" (UID: \"4c2ebad9-de50-4346-a64f-6cceecf44434\") " pod="openshift-dns/node-resolver-npnjm" Jul 27 21:03:30.237049 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.235376 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1907dc14-1132-4eb9-a5ee-675b43df6e6d-log-socket\") pod \"ovnkube-node-l6mhn\" (UID: \"1907dc14-1132-4eb9-a5ee-675b43df6e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" Jul 27 21:03:30.237049 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.235406 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1907dc14-1132-4eb9-a5ee-675b43df6e6d-ovn-node-metrics-cert\") pod \"ovnkube-node-l6mhn\" (UID: \"1907dc14-1132-4eb9-a5ee-675b43df6e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" Jul 27 21:03:30.237049 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.235432 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6f4b2b00-fd6c-463e-b27a-e1a925557c2f-sys\") pod \"tuned-hq8kb\" (UID: \"6f4b2b00-fd6c-463e-b27a-e1a925557c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-hq8kb" Jul 27 21:03:30.237049 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.235457 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f4b2b00-fd6c-463e-b27a-e1a925557c2f-var-lib-kubelet\") pod \"tuned-hq8kb\" (UID: \"6f4b2b00-fd6c-463e-b27a-e1a925557c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-hq8kb" Jul 27 21:03:30.237049 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.235499 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8addb952-4638-47ca-a4fb-d04fb3441aa9-host-var-lib-cni-multus\") pod \"multus-tv2n5\" (UID: \"8addb952-4638-47ca-a4fb-d04fb3441aa9\") " pod="openshift-multus/multus-tv2n5" Jul 27 21:03:30.237049 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.235527 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8addb952-4638-47ca-a4fb-d04fb3441aa9-hostroot\") pod \"multus-tv2n5\" (UID: \"8addb952-4638-47ca-a4fb-d04fb3441aa9\") " pod="openshift-multus/multus-tv2n5" Jul 27 21:03:30.237049 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.235554 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1907dc14-1132-4eb9-a5ee-675b43df6e6d-var-lib-openvswitch\") pod \"ovnkube-node-l6mhn\" (UID: \"1907dc14-1132-4eb9-a5ee-675b43df6e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" Jul 27 21:03:30.237049 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.235582 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1907dc14-1132-4eb9-a5ee-675b43df6e6d-run-openvswitch\") pod \"ovnkube-node-l6mhn\" (UID: \"1907dc14-1132-4eb9-a5ee-675b43df6e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" Jul 27 21:03:30.237049 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.235608 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1907dc14-1132-4eb9-a5ee-675b43df6e6d-env-overrides\") pod \"ovnkube-node-l6mhn\" (UID: \"1907dc14-1132-4eb9-a5ee-675b43df6e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" Jul 27 21:03:30.237049 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.235636 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1907dc14-1132-4eb9-a5ee-675b43df6e6d-host-cni-bin\") pod \"ovnkube-node-l6mhn\" (UID: \"1907dc14-1132-4eb9-a5ee-675b43df6e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" Jul 27 21:03:30.237049 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.235687 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1907dc14-1132-4eb9-a5ee-675b43df6e6d-host-cni-netd\") pod \"ovnkube-node-l6mhn\" (UID: \"1907dc14-1132-4eb9-a5ee-675b43df6e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" Jul 27 21:03:30.237049 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.235711 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8addb952-4638-47ca-a4fb-d04fb3441aa9-system-cni-dir\") pod \"multus-tv2n5\" (UID: \"8addb952-4638-47ca-a4fb-d04fb3441aa9\") " pod="openshift-multus/multus-tv2n5" Jul 27 21:03:30.237049 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.235744 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8addb952-4638-47ca-a4fb-d04fb3441aa9-etc-kubernetes\") pod \"multus-tv2n5\" (UID: \"8addb952-4638-47ca-a4fb-d04fb3441aa9\") " pod="openshift-multus/multus-tv2n5" Jul 27 21:03:30.237049 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.235771 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/de32d9f8-5f7d-4a68-8aa4-2830c11db8f5-konnectivity-ca\") pod \"konnectivity-agent-tx4cs\" (UID: \"de32d9f8-5f7d-4a68-8aa4-2830c11db8f5\") " pod="kube-system/konnectivity-agent-tx4cs" Jul 27 21:03:30.237996 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.235800 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v82f\" (UniqueName: \"kubernetes.io/projected/f8ef5264-95a3-4193-ae7d-f0f10a7733ab-kube-api-access-5v82f\") pod \"iptables-alerter-64nks\" (UID: \"f8ef5264-95a3-4193-ae7d-f0f10a7733ab\") " pod="openshift-network-operator/iptables-alerter-64nks" Jul 27 21:03:30.237996 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.235827 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1907dc14-1132-4eb9-a5ee-675b43df6e6d-systemd-units\") pod \"ovnkube-node-l6mhn\" (UID: \"1907dc14-1132-4eb9-a5ee-675b43df6e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" Jul 27 21:03:30.237996 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.235852 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1907dc14-1132-4eb9-a5ee-675b43df6e6d-node-log\") pod \"ovnkube-node-l6mhn\" (UID: \"1907dc14-1132-4eb9-a5ee-675b43df6e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" Jul 27 21:03:30.237996 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.235893 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8addb952-4638-47ca-a4fb-d04fb3441aa9-multus-conf-dir\") pod \"multus-tv2n5\" (UID: \"8addb952-4638-47ca-a4fb-d04fb3441aa9\") " pod="openshift-multus/multus-tv2n5" Jul 27 21:03:30.237996 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.235953 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8addb952-4638-47ca-a4fb-d04fb3441aa9-multus-daemon-config\") pod \"multus-tv2n5\" (UID: \"8addb952-4638-47ca-a4fb-d04fb3441aa9\") " pod="openshift-multus/multus-tv2n5" Jul 27 21:03:30.237996 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.235982 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1907dc14-1132-4eb9-a5ee-675b43df6e6d-run-ovn\") pod \"ovnkube-node-l6mhn\" (UID: \"1907dc14-1132-4eb9-a5ee-675b43df6e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" Jul 27 21:03:30.237996 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.236034 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1907dc14-1132-4eb9-a5ee-675b43df6e6d-host-run-ovn-kubernetes\") pod \"ovnkube-node-l6mhn\" (UID: \"1907dc14-1132-4eb9-a5ee-675b43df6e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" Jul 27 21:03:30.237996 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.236096 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6f4b2b00-fd6c-463e-b27a-e1a925557c2f-etc-sysconfig\") pod \"tuned-hq8kb\" (UID: \"6f4b2b00-fd6c-463e-b27a-e1a925557c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-hq8kb" Jul 27 21:03:30.237996 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.236119 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6f4b2b00-fd6c-463e-b27a-e1a925557c2f-etc-sysctl-d\") pod \"tuned-hq8kb\" (UID: \"6f4b2b00-fd6c-463e-b27a-e1a925557c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-hq8kb" Jul 27 21:03:30.237996 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.236139 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6f4b2b00-fd6c-463e-b27a-e1a925557c2f-run\") pod \"tuned-hq8kb\" (UID: \"6f4b2b00-fd6c-463e-b27a-e1a925557c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-hq8kb" Jul 27 21:03:30.237996 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.236156 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m52wq\" (UniqueName: \"kubernetes.io/projected/e7bf9687-782f-40c9-bb73-e8b4b2c6a763-kube-api-access-m52wq\") pod \"node-ca-zt59k\" (UID: \"e7bf9687-782f-40c9-bb73-e8b4b2c6a763\") " pod="openshift-image-registry/node-ca-zt59k" Jul 27 21:03:30.237996 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.236171 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1907dc14-1132-4eb9-a5ee-675b43df6e6d-host-run-netns\") pod \"ovnkube-node-l6mhn\" (UID: \"1907dc14-1132-4eb9-a5ee-675b43df6e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" Jul 27 21:03:30.237996 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.236187 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8addb952-4638-47ca-a4fb-d04fb3441aa9-host-run-k8s-cni-cncf-io\") pod \"multus-tv2n5\" (UID: \"8addb952-4638-47ca-a4fb-d04fb3441aa9\") " pod="openshift-multus/multus-tv2n5" Jul 27 21:03:30.237996 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.236227 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8addb952-4638-47ca-a4fb-d04fb3441aa9-host-var-lib-kubelet\") pod \"multus-tv2n5\" (UID: \"8addb952-4638-47ca-a4fb-d04fb3441aa9\") " pod="openshift-multus/multus-tv2n5" Jul 27 21:03:30.237996 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.236273 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c77555ba-462e-49c5-b8b0-e83987e8a2e3-kubelet-config\") pod \"global-pull-secret-syncer-l2czx\" (UID: \"c77555ba-462e-49c5-b8b0-e83987e8a2e3\") " pod="kube-system/global-pull-secret-syncer-l2czx" Jul 27 21:03:30.237996 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.236295 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6f4b2b00-fd6c-463e-b27a-e1a925557c2f-host\") pod \"tuned-hq8kb\" (UID: \"6f4b2b00-fd6c-463e-b27a-e1a925557c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-hq8kb" Jul 27 21:03:30.238859 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.236330 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1907dc14-1132-4eb9-a5ee-675b43df6e6d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l6mhn\" (UID: \"1907dc14-1132-4eb9-a5ee-675b43df6e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" Jul 27 21:03:30.238859 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.236351 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1907dc14-1132-4eb9-a5ee-675b43df6e6d-ovnkube-script-lib\") pod \"ovnkube-node-l6mhn\" (UID: \"1907dc14-1132-4eb9-a5ee-675b43df6e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" Jul 27 21:03:30.238859 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.236400 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f4b2b00-fd6c-463e-b27a-e1a925557c2f-etc-kubernetes\") pod \"tuned-hq8kb\" (UID: \"6f4b2b00-fd6c-463e-b27a-e1a925557c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-hq8kb" Jul 27 21:03:30.238859 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.236988 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 27 21:03:30.238859 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.237106 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 27 21:03:30.238859 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.237128 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 27 21:03:30.238859 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.237258 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 27 21:03:30.238859 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.237377 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-trrwd\"" Jul 27 21:03:30.238859 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.237387 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xc7tp\"" Jul 27 21:03:30.238859 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.237432 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5kfcg" Jul 27 21:03:30.238859 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.237448 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 27 21:03:30.238859 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.237936 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 27 21:03:30.240260 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.240077 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6k78d\"" Jul 27 21:03:30.240260 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.240085 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 27 21:03:30.240260 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.240150 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 27 21:03:30.242246 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.242222 2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 27 21:03:30.260587 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.260461 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-vvmz5" Jul 27 21:03:30.268891 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.268863 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-vvmz5" Jul 27 21:03:30.322788 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.322753 2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 27 21:03:30.336954 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.336910 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjdsl\" (UniqueName: \"kubernetes.io/projected/1907dc14-1132-4eb9-a5ee-675b43df6e6d-kube-api-access-bjdsl\") pod \"ovnkube-node-l6mhn\" (UID: \"1907dc14-1132-4eb9-a5ee-675b43df6e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" Jul 27 21:03:30.337165 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.336963 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6f4b2b00-fd6c-463e-b27a-e1a925557c2f-etc-sysctl-conf\") pod \"tuned-hq8kb\" (UID: \"6f4b2b00-fd6c-463e-b27a-e1a925557c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-hq8kb" Jul 27 21:03:30.337165 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.336998 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bbf44a86-6e26-4584-b851-86bcbf8d10f9-system-cni-dir\") pod \"multus-additional-cni-plugins-5kfcg\" (UID: \"bbf44a86-6e26-4584-b851-86bcbf8d10f9\") " pod="openshift-multus/multus-additional-cni-plugins-5kfcg" Jul 27 21:03:30.337165 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.337030 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bbf44a86-6e26-4584-b851-86bcbf8d10f9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5kfcg\" (UID: \"bbf44a86-6e26-4584-b851-86bcbf8d10f9\") " pod="openshift-multus/multus-additional-cni-plugins-5kfcg" Jul 27 21:03:30.337165 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.337057 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8addb952-4638-47ca-a4fb-d04fb3441aa9-multus-cni-dir\") pod \"multus-tv2n5\" (UID: \"8addb952-4638-47ca-a4fb-d04fb3441aa9\") " pod="openshift-multus/multus-tv2n5" Jul 27 21:03:30.337165 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.337083 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e7bf9687-782f-40c9-bb73-e8b4b2c6a763-serviceca\") pod \"node-ca-zt59k\" (UID: \"e7bf9687-782f-40c9-bb73-e8b4b2c6a763\") " pod="openshift-image-registry/node-ca-zt59k" Jul 27 21:03:30.337165 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.337112 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c77555ba-462e-49c5-b8b0-e83987e8a2e3-original-pull-secret\") pod \"global-pull-secret-syncer-l2czx\" (UID: \"c77555ba-462e-49c5-b8b0-e83987e8a2e3\") " pod="kube-system/global-pull-secret-syncer-l2czx" Jul 27 21:03:30.337165 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.337135 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6f4b2b00-fd6c-463e-b27a-e1a925557c2f-etc-tuned\") pod \"tuned-hq8kb\" (UID: \"6f4b2b00-fd6c-463e-b27a-e1a925557c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-hq8kb" Jul 27 21:03:30.337165 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.337162 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj9pc\" (UniqueName: \"kubernetes.io/projected/2f0a4058-4490-42ed-ae54-82e7ab83d15d-kube-api-access-jj9pc\") pod \"aws-ebs-csi-driver-node-jz7ws\" (UID: \"2f0a4058-4490-42ed-ae54-82e7ab83d15d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jz7ws" Jul 27 21:03:30.337591 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.337189 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bbf44a86-6e26-4584-b851-86bcbf8d10f9-os-release\") pod \"multus-additional-cni-plugins-5kfcg\" (UID: \"bbf44a86-6e26-4584-b851-86bcbf8d10f9\") " pod="openshift-multus/multus-additional-cni-plugins-5kfcg" Jul 27 21:03:30.337591 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.337201 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6f4b2b00-fd6c-463e-b27a-e1a925557c2f-etc-sysctl-conf\") pod \"tuned-hq8kb\" (UID: \"6f4b2b00-fd6c-463e-b27a-e1a925557c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-hq8kb" Jul 27 21:03:30.337591 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.337218 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bbf44a86-6e26-4584-b851-86bcbf8d10f9-cni-binary-copy\") pod \"multus-additional-cni-plugins-5kfcg\" (UID: \"bbf44a86-6e26-4584-b851-86bcbf8d10f9\") " pod="openshift-multus/multus-additional-cni-plugins-5kfcg" Jul 27 21:03:30.337591 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.337265 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2f0a4058-4490-42ed-ae54-82e7ab83d15d-device-dir\") pod \"aws-ebs-csi-driver-node-jz7ws\" (UID: \"2f0a4058-4490-42ed-ae54-82e7ab83d15d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jz7ws" Jul 27 21:03:30.337591 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.337298 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8addb952-4638-47ca-a4fb-d04fb3441aa9-host-run-multus-certs\") pod \"multus-tv2n5\" (UID: \"8addb952-4638-47ca-a4fb-d04fb3441aa9\") " pod="openshift-multus/multus-tv2n5" Jul 27 21:03:30.337591 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.337315 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e7bf9687-782f-40c9-bb73-e8b4b2c6a763-host\") pod \"node-ca-zt59k\" (UID: \"e7bf9687-782f-40c9-bb73-e8b4b2c6a763\") " pod="openshift-image-registry/node-ca-zt59k" Jul 27 21:03:30.337591 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.337318 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8addb952-4638-47ca-a4fb-d04fb3441aa9-multus-cni-dir\") pod \"multus-tv2n5\" (UID: \"8addb952-4638-47ca-a4fb-d04fb3441aa9\") " pod="openshift-multus/multus-tv2n5" Jul 27 21:03:30.337591 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.337334 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c32154a-b41d-45c4-afa6-240a8ea24835-metrics-certs\") pod \"network-metrics-daemon-lvg8z\" (UID: \"0c32154a-b41d-45c4-afa6-240a8ea24835\") " pod="openshift-multus/network-metrics-daemon-lvg8z" Jul 27 21:03:30.337591 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:30.337345 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 21:03:30.337591 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.337359 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft79h\" (UniqueName: \"kubernetes.io/projected/8157bc07-d223-44de-aee6-ffa0c683a2cb-kube-api-access-ft79h\") pod \"network-check-target-lrnv5\" (UID: \"8157bc07-d223-44de-aee6-ffa0c683a2cb\") " pod="openshift-network-diagnostics/network-check-target-lrnv5" Jul 27 21:03:30.337591 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.337378 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c77555ba-462e-49c5-b8b0-e83987e8a2e3-dbus\") pod \"global-pull-secret-syncer-l2czx\" (UID: \"c77555ba-462e-49c5-b8b0-e83987e8a2e3\") " pod="kube-system/global-pull-secret-syncer-l2czx" Jul 27 21:03:30.337591 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.337394 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f8ef5264-95a3-4193-ae7d-f0f10a7733ab-host-slash\") pod \"iptables-alerter-64nks\" (UID: \"f8ef5264-95a3-4193-ae7d-f0f10a7733ab\") " pod="openshift-network-operator/iptables-alerter-64nks" Jul 27 21:03:30.337591 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.337428 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f8ef5264-95a3-4193-ae7d-f0f10a7733ab-host-slash\") pod \"iptables-alerter-64nks\" (UID: \"f8ef5264-95a3-4193-ae7d-f0f10a7733ab\") " pod="openshift-network-operator/iptables-alerter-64nks" Jul 27 21:03:30.337591 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:30.337449 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c77555ba-462e-49c5-b8b0-e83987e8a2e3-original-pull-secret podName:c77555ba-462e-49c5-b8b0-e83987e8a2e3 nodeName:}" failed. No retries permitted until 2026-07-27 21:03:30.837399654 +0000 UTC m=+2.969015871 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c77555ba-462e-49c5-b8b0-e83987e8a2e3-original-pull-secret") pod "global-pull-secret-syncer-l2czx" (UID: "c77555ba-462e-49c5-b8b0-e83987e8a2e3") : object "kube-system"/"original-pull-secret" not registered Jul 27 21:03:30.337591 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.337460 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8addb952-4638-47ca-a4fb-d04fb3441aa9-host-run-multus-certs\") pod \"multus-tv2n5\" (UID: \"8addb952-4638-47ca-a4fb-d04fb3441aa9\") " pod="openshift-multus/multus-tv2n5" Jul 27 21:03:30.337591 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.337503 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e7bf9687-782f-40c9-bb73-e8b4b2c6a763-host\") pod \"node-ca-zt59k\" (UID: \"e7bf9687-782f-40c9-bb73-e8b4b2c6a763\") " pod="openshift-image-registry/node-ca-zt59k" Jul 27 21:03:30.337591 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:30.337595 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 21:03:30.338539 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:30.337655 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c32154a-b41d-45c4-afa6-240a8ea24835-metrics-certs podName:0c32154a-b41d-45c4-afa6-240a8ea24835 nodeName:}" failed. No retries permitted until 2026-07-27 21:03:30.837623866 +0000 UTC m=+2.969240084 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0c32154a-b41d-45c4-afa6-240a8ea24835-metrics-certs") pod "network-metrics-daemon-lvg8z" (UID: "0c32154a-b41d-45c4-afa6-240a8ea24835") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 21:03:30.338539 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.337683 2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 27 21:03:30.338539 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.337702 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fsh5\" (UniqueName: \"kubernetes.io/projected/6f4b2b00-fd6c-463e-b27a-e1a925557c2f-kube-api-access-8fsh5\") pod \"tuned-hq8kb\" (UID: \"6f4b2b00-fd6c-463e-b27a-e1a925557c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-hq8kb" Jul 27 21:03:30.338539 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.337690 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e7bf9687-782f-40c9-bb73-e8b4b2c6a763-serviceca\") pod \"node-ca-zt59k\" (UID: \"e7bf9687-782f-40c9-bb73-e8b4b2c6a763\") " pod="openshift-image-registry/node-ca-zt59k" Jul 27 21:03:30.338539 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.337894 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c77555ba-462e-49c5-b8b0-e83987e8a2e3-dbus\") pod \"global-pull-secret-syncer-l2czx\" (UID: \"c77555ba-462e-49c5-b8b0-e83987e8a2e3\") " pod="kube-system/global-pull-secret-syncer-l2czx" Jul 27 21:03:30.338539 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.337932 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bbf44a86-6e26-4584-b851-86bcbf8d10f9-cnibin\") pod \"multus-additional-cni-plugins-5kfcg\" (UID: \"bbf44a86-6e26-4584-b851-86bcbf8d10f9\") " pod="openshift-multus/multus-additional-cni-plugins-5kfcg" Jul 27 21:03:30.338539 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.337961 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8addb952-4638-47ca-a4fb-d04fb3441aa9-os-release\") pod \"multus-tv2n5\" (UID: \"8addb952-4638-47ca-a4fb-d04fb3441aa9\") " pod="openshift-multus/multus-tv2n5" Jul 27 21:03:30.338539 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.337990 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f8ef5264-95a3-4193-ae7d-f0f10a7733ab-iptables-alerter-script\") pod \"iptables-alerter-64nks\" (UID: \"f8ef5264-95a3-4193-ae7d-f0f10a7733ab\") " pod="openshift-network-operator/iptables-alerter-64nks" Jul 27 21:03:30.338539 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.338019 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6f4b2b00-fd6c-463e-b27a-e1a925557c2f-etc-modprobe-d\") pod \"tuned-hq8kb\" (UID: \"6f4b2b00-fd6c-463e-b27a-e1a925557c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-hq8kb" Jul 27 21:03:30.338539 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.338045 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2f0a4058-4490-42ed-ae54-82e7ab83d15d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jz7ws\" (UID: \"2f0a4058-4490-42ed-ae54-82e7ab83d15d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jz7ws" Jul 27 21:03:30.338539 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.338076 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8addb952-4638-47ca-a4fb-d04fb3441aa9-os-release\") pod \"multus-tv2n5\" (UID: \"8addb952-4638-47ca-a4fb-d04fb3441aa9\") " pod="openshift-multus/multus-tv2n5" Jul 27 21:03:30.338539 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.338115 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bbf44a86-6e26-4584-b851-86bcbf8d10f9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5kfcg\" (UID: \"bbf44a86-6e26-4584-b851-86bcbf8d10f9\") " pod="openshift-multus/multus-additional-cni-plugins-5kfcg" Jul 27 21:03:30.338539 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.338150 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlhxq\" (UniqueName: \"kubernetes.io/projected/bbf44a86-6e26-4584-b851-86bcbf8d10f9-kube-api-access-dlhxq\") pod \"multus-additional-cni-plugins-5kfcg\" (UID: \"bbf44a86-6e26-4584-b851-86bcbf8d10f9\") " pod="openshift-multus/multus-additional-cni-plugins-5kfcg" Jul 27 21:03:30.338539 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.338176 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/de32d9f8-5f7d-4a68-8aa4-2830c11db8f5-agent-certs\") pod \"konnectivity-agent-tx4cs\" (UID: \"de32d9f8-5f7d-4a68-8aa4-2830c11db8f5\") " pod="kube-system/konnectivity-agent-tx4cs" Jul 27 21:03:30.338539 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.338194 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6f4b2b00-fd6c-463e-b27a-e1a925557c2f-etc-modprobe-d\") pod \"tuned-hq8kb\" (UID: \"6f4b2b00-fd6c-463e-b27a-e1a925557c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-hq8kb" Jul 27 21:03:30.338539 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.338205 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8addb952-4638-47ca-a4fb-d04fb3441aa9-multus-socket-dir-parent\") pod \"multus-tv2n5\" (UID: \"8addb952-4638-47ca-a4fb-d04fb3441aa9\") " pod="openshift-multus/multus-tv2n5" Jul 27 21:03:30.338539 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.338242 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8addb952-4638-47ca-a4fb-d04fb3441aa9-host-var-lib-cni-bin\") pod \"multus-tv2n5\" (UID: \"8addb952-4638-47ca-a4fb-d04fb3441aa9\") " pod="openshift-multus/multus-tv2n5" Jul 27 21:03:30.339355 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.338261 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8addb952-4638-47ca-a4fb-d04fb3441aa9-multus-socket-dir-parent\") pod \"multus-tv2n5\" (UID: \"8addb952-4638-47ca-a4fb-d04fb3441aa9\") " pod="openshift-multus/multus-tv2n5" Jul 27 21:03:30.339355 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.338269 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6f4b2b00-fd6c-463e-b27a-e1a925557c2f-etc-systemd\") pod \"tuned-hq8kb\" (UID: \"6f4b2b00-fd6c-463e-b27a-e1a925557c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-hq8kb" Jul 27 21:03:30.339355 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.338303 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2f0a4058-4490-42ed-ae54-82e7ab83d15d-etc-selinux\") pod \"aws-ebs-csi-driver-node-jz7ws\" (UID: \"2f0a4058-4490-42ed-ae54-82e7ab83d15d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jz7ws" Jul 27 21:03:30.339355 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.338319 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8addb952-4638-47ca-a4fb-d04fb3441aa9-host-var-lib-cni-bin\") pod \"multus-tv2n5\" (UID: \"8addb952-4638-47ca-a4fb-d04fb3441aa9\") " pod="openshift-multus/multus-tv2n5" Jul 27 21:03:30.339355 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.338328 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4c2ebad9-de50-4346-a64f-6cceecf44434-hosts-file\") pod \"node-resolver-npnjm\" (UID: \"4c2ebad9-de50-4346-a64f-6cceecf44434\") " pod="openshift-dns/node-resolver-npnjm" Jul 27 21:03:30.339355 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.338349 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1907dc14-1132-4eb9-a5ee-675b43df6e6d-log-socket\") pod \"ovnkube-node-l6mhn\" (UID: \"1907dc14-1132-4eb9-a5ee-675b43df6e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" Jul 27 21:03:30.339355 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.338389 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1907dc14-1132-4eb9-a5ee-675b43df6e6d-log-socket\") pod \"ovnkube-node-l6mhn\" (UID: \"1907dc14-1132-4eb9-a5ee-675b43df6e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" Jul 27 21:03:30.339355 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.338427 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6f4b2b00-fd6c-463e-b27a-e1a925557c2f-etc-systemd\") pod \"tuned-hq8kb\" (UID: \"6f4b2b00-fd6c-463e-b27a-e1a925557c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-hq8kb" Jul 27 21:03:30.339355 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.338468 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4c2ebad9-de50-4346-a64f-6cceecf44434-hosts-file\") pod \"node-resolver-npnjm\" (UID: \"4c2ebad9-de50-4346-a64f-6cceecf44434\") " pod="openshift-dns/node-resolver-npnjm" Jul 27 21:03:30.339355 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.338509 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1907dc14-1132-4eb9-a5ee-675b43df6e6d-ovn-node-metrics-cert\") pod \"ovnkube-node-l6mhn\" (UID: \"1907dc14-1132-4eb9-a5ee-675b43df6e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" Jul 27 21:03:30.339355 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.338532 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6f4b2b00-fd6c-463e-b27a-e1a925557c2f-sys\") pod \"tuned-hq8kb\" (UID: \"6f4b2b00-fd6c-463e-b27a-e1a925557c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-hq8kb" Jul 27 21:03:30.339355 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.338554 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f4b2b00-fd6c-463e-b27a-e1a925557c2f-var-lib-kubelet\") pod \"tuned-hq8kb\" (UID: \"6f4b2b00-fd6c-463e-b27a-e1a925557c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-hq8kb" Jul 27 21:03:30.339355 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.338576 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bbf44a86-6e26-4584-b851-86bcbf8d10f9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5kfcg\" (UID: \"bbf44a86-6e26-4584-b851-86bcbf8d10f9\") " pod="openshift-multus/multus-additional-cni-plugins-5kfcg" Jul 27 21:03:30.339355 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.338604 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8addb952-4638-47ca-a4fb-d04fb3441aa9-host-var-lib-cni-multus\") pod \"multus-tv2n5\" (UID: \"8addb952-4638-47ca-a4fb-d04fb3441aa9\") " pod="openshift-multus/multus-tv2n5" Jul 27 21:03:30.339355 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.338639 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f8ef5264-95a3-4193-ae7d-f0f10a7733ab-iptables-alerter-script\") pod \"iptables-alerter-64nks\" (UID: \"f8ef5264-95a3-4193-ae7d-f0f10a7733ab\") " pod="openshift-network-operator/iptables-alerter-64nks" Jul 27 21:03:30.339355 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.338647 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8addb952-4638-47ca-a4fb-d04fb3441aa9-hostroot\") pod \"multus-tv2n5\" (UID: \"8addb952-4638-47ca-a4fb-d04fb3441aa9\") " pod="openshift-multus/multus-tv2n5" Jul 27 21:03:30.339355 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.338705 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1907dc14-1132-4eb9-a5ee-675b43df6e6d-var-lib-openvswitch\") pod \"ovnkube-node-l6mhn\" (UID: \"1907dc14-1132-4eb9-a5ee-675b43df6e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" Jul 27 21:03:30.340251 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.338718 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f4b2b00-fd6c-463e-b27a-e1a925557c2f-var-lib-kubelet\") pod \"tuned-hq8kb\" (UID: \"6f4b2b00-fd6c-463e-b27a-e1a925557c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-hq8kb" Jul 27 21:03:30.340251 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.338733 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1907dc14-1132-4eb9-a5ee-675b43df6e6d-run-openvswitch\") pod \"ovnkube-node-l6mhn\" (UID: \"1907dc14-1132-4eb9-a5ee-675b43df6e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" Jul 27 21:03:30.340251 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.338786 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8addb952-4638-47ca-a4fb-d04fb3441aa9-hostroot\") pod \"multus-tv2n5\" (UID: \"8addb952-4638-47ca-a4fb-d04fb3441aa9\") " pod="openshift-multus/multus-tv2n5" Jul 27 21:03:30.340251 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.338794 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1907dc14-1132-4eb9-a5ee-675b43df6e6d-run-openvswitch\") pod \"ovnkube-node-l6mhn\" (UID: \"1907dc14-1132-4eb9-a5ee-675b43df6e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" Jul 27 21:03:30.340251 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.338821 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1907dc14-1132-4eb9-a5ee-675b43df6e6d-var-lib-openvswitch\") pod \"ovnkube-node-l6mhn\" (UID: \"1907dc14-1132-4eb9-a5ee-675b43df6e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" Jul 27 21:03:30.340251 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.338859 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8addb952-4638-47ca-a4fb-d04fb3441aa9-host-var-lib-cni-multus\") pod \"multus-tv2n5\" (UID: \"8addb952-4638-47ca-a4fb-d04fb3441aa9\") " pod="openshift-multus/multus-tv2n5" Jul 27 21:03:30.340251 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.338887 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1907dc14-1132-4eb9-a5ee-675b43df6e6d-env-overrides\") pod \"ovnkube-node-l6mhn\" (UID: \"1907dc14-1132-4eb9-a5ee-675b43df6e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" Jul 27 21:03:30.340251 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.338919 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1907dc14-1132-4eb9-a5ee-675b43df6e6d-host-cni-bin\") pod \"ovnkube-node-l6mhn\" (UID: \"1907dc14-1132-4eb9-a5ee-675b43df6e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" Jul 27 21:03:30.340251 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.338946 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1907dc14-1132-4eb9-a5ee-675b43df6e6d-host-cni-netd\") pod \"ovnkube-node-l6mhn\" (UID: \"1907dc14-1132-4eb9-a5ee-675b43df6e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" Jul 27 21:03:30.340251 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.338972 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8addb952-4638-47ca-a4fb-d04fb3441aa9-system-cni-dir\") pod \"multus-tv2n5\" (UID: \"8addb952-4638-47ca-a4fb-d04fb3441aa9\") " pod="openshift-multus/multus-tv2n5" Jul 27 21:03:30.340251 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.338977 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1907dc14-1132-4eb9-a5ee-675b43df6e6d-host-cni-bin\") pod \"ovnkube-node-l6mhn\" (UID: \"1907dc14-1132-4eb9-a5ee-675b43df6e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" Jul 27 21:03:30.340251 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.338947 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6f4b2b00-fd6c-463e-b27a-e1a925557c2f-sys\") pod \"tuned-hq8kb\" (UID: \"6f4b2b00-fd6c-463e-b27a-e1a925557c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-hq8kb" Jul 27 21:03:30.340251 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.338998 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8addb952-4638-47ca-a4fb-d04fb3441aa9-etc-kubernetes\") pod \"multus-tv2n5\" (UID: \"8addb952-4638-47ca-a4fb-d04fb3441aa9\") " pod="openshift-multus/multus-tv2n5" Jul 27 21:03:30.340251 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.339025 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/de32d9f8-5f7d-4a68-8aa4-2830c11db8f5-konnectivity-ca\") pod \"konnectivity-agent-tx4cs\" (UID: \"de32d9f8-5f7d-4a68-8aa4-2830c11db8f5\") " pod="kube-system/konnectivity-agent-tx4cs" Jul 27 21:03:30.340251 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.339022 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1907dc14-1132-4eb9-a5ee-675b43df6e6d-host-cni-netd\") pod \"ovnkube-node-l6mhn\" (UID: \"1907dc14-1132-4eb9-a5ee-675b43df6e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" Jul 27 21:03:30.340251 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.339057 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v82f\" (UniqueName: \"kubernetes.io/projected/f8ef5264-95a3-4193-ae7d-f0f10a7733ab-kube-api-access-5v82f\") pod \"iptables-alerter-64nks\" (UID: \"f8ef5264-95a3-4193-ae7d-f0f10a7733ab\") " pod="openshift-network-operator/iptables-alerter-64nks" Jul 27 21:03:30.340251 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.339082 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1907dc14-1132-4eb9-a5ee-675b43df6e6d-systemd-units\") pod \"ovnkube-node-l6mhn\" (UID: \"1907dc14-1132-4eb9-a5ee-675b43df6e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" Jul 27 21:03:30.340251 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.339086 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8addb952-4638-47ca-a4fb-d04fb3441aa9-system-cni-dir\") pod \"multus-tv2n5\" (UID: \"8addb952-4638-47ca-a4fb-d04fb3441aa9\") " pod="openshift-multus/multus-tv2n5" Jul 27 21:03:30.341162 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.339105 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1907dc14-1132-4eb9-a5ee-675b43df6e6d-node-log\") pod \"ovnkube-node-l6mhn\" (UID: \"1907dc14-1132-4eb9-a5ee-675b43df6e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" Jul 27 21:03:30.341162 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.339088 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8addb952-4638-47ca-a4fb-d04fb3441aa9-etc-kubernetes\") pod \"multus-tv2n5\" (UID: \"8addb952-4638-47ca-a4fb-d04fb3441aa9\") " pod="openshift-multus/multus-tv2n5" Jul 27 21:03:30.341162 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.339157 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8addb952-4638-47ca-a4fb-d04fb3441aa9-multus-conf-dir\") pod \"multus-tv2n5\" (UID: \"8addb952-4638-47ca-a4fb-d04fb3441aa9\") " pod="openshift-multus/multus-tv2n5" Jul 27 21:03:30.341162 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.339192 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8addb952-4638-47ca-a4fb-d04fb3441aa9-multus-daemon-config\") pod \"multus-tv2n5\" (UID: \"8addb952-4638-47ca-a4fb-d04fb3441aa9\") " pod="openshift-multus/multus-tv2n5" Jul 27 21:03:30.341162 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.339198 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1907dc14-1132-4eb9-a5ee-675b43df6e6d-systemd-units\") pod \"ovnkube-node-l6mhn\" (UID: \"1907dc14-1132-4eb9-a5ee-675b43df6e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" Jul 27 21:03:30.341162 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.339221 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1907dc14-1132-4eb9-a5ee-675b43df6e6d-run-ovn\") pod \"ovnkube-node-l6mhn\" (UID: \"1907dc14-1132-4eb9-a5ee-675b43df6e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" Jul 27 21:03:30.341162 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.339252 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1907dc14-1132-4eb9-a5ee-675b43df6e6d-host-run-ovn-kubernetes\") pod \"ovnkube-node-l6mhn\" (UID: \"1907dc14-1132-4eb9-a5ee-675b43df6e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" Jul 27 21:03:30.341162 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.339279 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6f4b2b00-fd6c-463e-b27a-e1a925557c2f-etc-sysconfig\") pod \"tuned-hq8kb\" (UID: \"6f4b2b00-fd6c-463e-b27a-e1a925557c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-hq8kb" Jul 27 21:03:30.341162 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.339333 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6f4b2b00-fd6c-463e-b27a-e1a925557c2f-etc-sysctl-d\") pod \"tuned-hq8kb\" (UID: \"6f4b2b00-fd6c-463e-b27a-e1a925557c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-hq8kb" Jul 27 21:03:30.341162 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.339351 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1907dc14-1132-4eb9-a5ee-675b43df6e6d-env-overrides\") pod \"ovnkube-node-l6mhn\" (UID: \"1907dc14-1132-4eb9-a5ee-675b43df6e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" Jul 27 21:03:30.341162 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.339362 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6f4b2b00-fd6c-463e-b27a-e1a925557c2f-run\") pod \"tuned-hq8kb\" (UID: \"6f4b2b00-fd6c-463e-b27a-e1a925557c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-hq8kb" Jul 27 21:03:30.341162 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.339409 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m52wq\" (UniqueName: \"kubernetes.io/projected/e7bf9687-782f-40c9-bb73-e8b4b2c6a763-kube-api-access-m52wq\") pod \"node-ca-zt59k\" (UID: \"e7bf9687-782f-40c9-bb73-e8b4b2c6a763\") " pod="openshift-image-registry/node-ca-zt59k" Jul 27 21:03:30.341162 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.339422 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8addb952-4638-47ca-a4fb-d04fb3441aa9-multus-conf-dir\") pod \"multus-tv2n5\" (UID: \"8addb952-4638-47ca-a4fb-d04fb3441aa9\") " pod="openshift-multus/multus-tv2n5" Jul 27 21:03:30.341162 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.339162 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1907dc14-1132-4eb9-a5ee-675b43df6e6d-node-log\") pod \"ovnkube-node-l6mhn\" (UID: \"1907dc14-1132-4eb9-a5ee-675b43df6e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" Jul 27 21:03:30.341162 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.339438 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1907dc14-1132-4eb9-a5ee-675b43df6e6d-host-run-netns\") pod \"ovnkube-node-l6mhn\" (UID: \"1907dc14-1132-4eb9-a5ee-675b43df6e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" Jul 27 21:03:30.341162 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.339474 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2f0a4058-4490-42ed-ae54-82e7ab83d15d-registration-dir\") pod \"aws-ebs-csi-driver-node-jz7ws\" (UID: \"2f0a4058-4490-42ed-ae54-82e7ab83d15d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jz7ws" Jul 27 21:03:30.341162 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.339487 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6f4b2b00-fd6c-463e-b27a-e1a925557c2f-etc-sysconfig\") pod \"tuned-hq8kb\" (UID: \"6f4b2b00-fd6c-463e-b27a-e1a925557c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-hq8kb" Jul 27 21:03:30.341162 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.339502 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2f0a4058-4490-42ed-ae54-82e7ab83d15d-sys-fs\") pod \"aws-ebs-csi-driver-node-jz7ws\" (UID: \"2f0a4058-4490-42ed-ae54-82e7ab83d15d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jz7ws" Jul 27 21:03:30.342105 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.339530 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1907dc14-1132-4eb9-a5ee-675b43df6e6d-run-ovn\") pod \"ovnkube-node-l6mhn\" (UID: \"1907dc14-1132-4eb9-a5ee-675b43df6e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" Jul 27 21:03:30.342105 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.339534 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8addb952-4638-47ca-a4fb-d04fb3441aa9-host-run-k8s-cni-cncf-io\") pod \"multus-tv2n5\" (UID: \"8addb952-4638-47ca-a4fb-d04fb3441aa9\") " pod="openshift-multus/multus-tv2n5" Jul 27 21:03:30.342105 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.339569 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8addb952-4638-47ca-a4fb-d04fb3441aa9-host-var-lib-kubelet\") pod \"multus-tv2n5\" (UID: \"8addb952-4638-47ca-a4fb-d04fb3441aa9\") " pod="openshift-multus/multus-tv2n5" Jul 27 21:03:30.342105 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.339586 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8addb952-4638-47ca-a4fb-d04fb3441aa9-host-run-k8s-cni-cncf-io\") pod \"multus-tv2n5\" (UID: \"8addb952-4638-47ca-a4fb-d04fb3441aa9\") " pod="openshift-multus/multus-tv2n5" Jul 27 21:03:30.342105 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.339596 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c77555ba-462e-49c5-b8b0-e83987e8a2e3-kubelet-config\") pod \"global-pull-secret-syncer-l2czx\" (UID: \"c77555ba-462e-49c5-b8b0-e83987e8a2e3\") " pod="kube-system/global-pull-secret-syncer-l2czx" Jul 27 21:03:30.342105 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.339620 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6f4b2b00-fd6c-463e-b27a-e1a925557c2f-host\") pod \"tuned-hq8kb\" (UID: \"6f4b2b00-fd6c-463e-b27a-e1a925557c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-hq8kb" Jul 27 21:03:30.342105 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.339666 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1907dc14-1132-4eb9-a5ee-675b43df6e6d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l6mhn\" (UID: \"1907dc14-1132-4eb9-a5ee-675b43df6e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" Jul 27 21:03:30.342105 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.339694 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1907dc14-1132-4eb9-a5ee-675b43df6e6d-ovnkube-script-lib\") pod \"ovnkube-node-l6mhn\" (UID: \"1907dc14-1132-4eb9-a5ee-675b43df6e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" Jul 27 21:03:30.342105 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.339716 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6f4b2b00-fd6c-463e-b27a-e1a925557c2f-etc-sysctl-d\") pod \"tuned-hq8kb\" (UID: \"6f4b2b00-fd6c-463e-b27a-e1a925557c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-hq8kb" Jul 27 21:03:30.342105 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.339720 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f4b2b00-fd6c-463e-b27a-e1a925557c2f-etc-kubernetes\") pod \"tuned-hq8kb\" (UID: \"6f4b2b00-fd6c-463e-b27a-e1a925557c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-hq8kb" Jul 27 21:03:30.342105 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.339763 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f4b2b00-fd6c-463e-b27a-e1a925557c2f-etc-kubernetes\") pod \"tuned-hq8kb\" (UID: \"6f4b2b00-fd6c-463e-b27a-e1a925557c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-hq8kb" Jul 27 21:03:30.342105 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.339769 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8addb952-4638-47ca-a4fb-d04fb3441aa9-cnibin\") pod \"multus-tv2n5\" (UID: \"8addb952-4638-47ca-a4fb-d04fb3441aa9\") " pod="openshift-multus/multus-tv2n5" Jul 27 21:03:30.342105 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.339803 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1907dc14-1132-4eb9-a5ee-675b43df6e6d-host-slash\") pod \"ovnkube-node-l6mhn\" (UID: \"1907dc14-1132-4eb9-a5ee-675b43df6e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" Jul 27 21:03:30.342105 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.339835 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1907dc14-1132-4eb9-a5ee-675b43df6e6d-host-run-netns\") pod \"ovnkube-node-l6mhn\" (UID: \"1907dc14-1132-4eb9-a5ee-675b43df6e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" Jul 27 21:03:30.342105 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.339840 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8addb952-4638-47ca-a4fb-d04fb3441aa9-host-var-lib-kubelet\") pod \"multus-tv2n5\" (UID: \"8addb952-4638-47ca-a4fb-d04fb3441aa9\") " pod="openshift-multus/multus-tv2n5" Jul 27 21:03:30.342105 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.339838 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/de32d9f8-5f7d-4a68-8aa4-2830c11db8f5-konnectivity-ca\") pod \"konnectivity-agent-tx4cs\" (UID: \"de32d9f8-5f7d-4a68-8aa4-2830c11db8f5\") " pod="kube-system/konnectivity-agent-tx4cs" Jul 27 21:03:30.342105 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.339852 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8addb952-4638-47ca-a4fb-d04fb3441aa9-multus-daemon-config\") pod \"multus-tv2n5\" (UID: \"8addb952-4638-47ca-a4fb-d04fb3441aa9\") " pod="openshift-multus/multus-tv2n5" Jul 27 21:03:30.342105 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.339882 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1907dc14-1132-4eb9-a5ee-675b43df6e6d-etc-openvswitch\") pod \"ovnkube-node-l6mhn\" (UID: \"1907dc14-1132-4eb9-a5ee-675b43df6e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" Jul 27 21:03:30.344511 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.339889 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6f4b2b00-fd6c-463e-b27a-e1a925557c2f-run\") pod \"tuned-hq8kb\" (UID: \"6f4b2b00-fd6c-463e-b27a-e1a925557c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-hq8kb" Jul 27 21:03:30.344511 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.339911 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1907dc14-1132-4eb9-a5ee-675b43df6e6d-host-slash\") pod \"ovnkube-node-l6mhn\" (UID: \"1907dc14-1132-4eb9-a5ee-675b43df6e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" Jul 27 21:03:30.344511 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.339919 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8addb952-4638-47ca-a4fb-d04fb3441aa9-cnibin\") pod \"multus-tv2n5\" (UID: \"8addb952-4638-47ca-a4fb-d04fb3441aa9\") " pod="openshift-multus/multus-tv2n5" Jul 27 21:03:30.344511 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.339806 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1907dc14-1132-4eb9-a5ee-675b43df6e6d-host-run-ovn-kubernetes\") pod \"ovnkube-node-l6mhn\" (UID: \"1907dc14-1132-4eb9-a5ee-675b43df6e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" Jul 27 21:03:30.344511 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.339933 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c77555ba-462e-49c5-b8b0-e83987e8a2e3-kubelet-config\") pod \"global-pull-secret-syncer-l2czx\" (UID: \"c77555ba-462e-49c5-b8b0-e83987e8a2e3\") " pod="kube-system/global-pull-secret-syncer-l2czx" Jul 27 21:03:30.344511 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.339840 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1907dc14-1132-4eb9-a5ee-675b43df6e6d-etc-openvswitch\") pod \"ovnkube-node-l6mhn\" (UID: \"1907dc14-1132-4eb9-a5ee-675b43df6e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" Jul 27 21:03:30.344511 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.339960 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6f4b2b00-fd6c-463e-b27a-e1a925557c2f-host\") pod \"tuned-hq8kb\" (UID: \"6f4b2b00-fd6c-463e-b27a-e1a925557c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-hq8kb" Jul 27 21:03:30.344511 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.339971 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1907dc14-1132-4eb9-a5ee-675b43df6e6d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l6mhn\" (UID: \"1907dc14-1132-4eb9-a5ee-675b43df6e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" Jul 27 21:03:30.344511 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.339972 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6f4b2b00-fd6c-463e-b27a-e1a925557c2f-lib-modules\") pod \"tuned-hq8kb\" (UID: \"6f4b2b00-fd6c-463e-b27a-e1a925557c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-hq8kb" Jul 27 21:03:30.344511 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.340015 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8addb952-4638-47ca-a4fb-d04fb3441aa9-cni-binary-copy\") pod \"multus-tv2n5\" (UID: \"8addb952-4638-47ca-a4fb-d04fb3441aa9\") " pod="openshift-multus/multus-tv2n5" Jul 27 21:03:30.344511 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.340047 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p2lc\" (UniqueName: \"kubernetes.io/projected/0c32154a-b41d-45c4-afa6-240a8ea24835-kube-api-access-7p2lc\") pod \"network-metrics-daemon-lvg8z\" (UID: \"0c32154a-b41d-45c4-afa6-240a8ea24835\") " pod="openshift-multus/network-metrics-daemon-lvg8z" Jul 27 21:03:30.344511 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.340055 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6f4b2b00-fd6c-463e-b27a-e1a925557c2f-lib-modules\") pod \"tuned-hq8kb\" (UID: \"6f4b2b00-fd6c-463e-b27a-e1a925557c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-hq8kb" Jul 27 21:03:30.344511 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.340076 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1907dc14-1132-4eb9-a5ee-675b43df6e6d-run-systemd\") pod \"ovnkube-node-l6mhn\" (UID: \"1907dc14-1132-4eb9-a5ee-675b43df6e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" Jul 27 21:03:30.344511 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.340104 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1907dc14-1132-4eb9-a5ee-675b43df6e6d-ovnkube-config\") pod \"ovnkube-node-l6mhn\" (UID: \"1907dc14-1132-4eb9-a5ee-675b43df6e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" Jul 27 21:03:30.344511 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.340125 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1907dc14-1132-4eb9-a5ee-675b43df6e6d-run-systemd\") pod \"ovnkube-node-l6mhn\" (UID: \"1907dc14-1132-4eb9-a5ee-675b43df6e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" Jul 27 21:03:30.344511 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.340133 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6f4b2b00-fd6c-463e-b27a-e1a925557c2f-tmp\") pod \"tuned-hq8kb\" (UID: \"6f4b2b00-fd6c-463e-b27a-e1a925557c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-hq8kb" Jul 27 21:03:30.344511 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.340162 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq4kc\" (UniqueName: \"kubernetes.io/projected/8addb952-4638-47ca-a4fb-d04fb3441aa9-kube-api-access-zq4kc\") pod \"multus-tv2n5\" (UID: \"8addb952-4638-47ca-a4fb-d04fb3441aa9\") " pod="openshift-multus/multus-tv2n5" Jul 27 21:03:30.344511 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.340191 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1907dc14-1132-4eb9-a5ee-675b43df6e6d-host-kubelet\") pod \"ovnkube-node-l6mhn\" (UID: \"1907dc14-1132-4eb9-a5ee-675b43df6e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" Jul 27 21:03:30.345472 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.340220 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2f0a4058-4490-42ed-ae54-82e7ab83d15d-socket-dir\") pod \"aws-ebs-csi-driver-node-jz7ws\" (UID: \"2f0a4058-4490-42ed-ae54-82e7ab83d15d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jz7ws" Jul 27 21:03:30.345472 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.340250 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8addb952-4638-47ca-a4fb-d04fb3441aa9-host-run-netns\") pod \"multus-tv2n5\" (UID: \"8addb952-4638-47ca-a4fb-d04fb3441aa9\") " pod="openshift-multus/multus-tv2n5" Jul 27 21:03:30.345472 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.340275 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c2ebad9-de50-4346-a64f-6cceecf44434-tmp-dir\") pod \"node-resolver-npnjm\" (UID: \"4c2ebad9-de50-4346-a64f-6cceecf44434\") " pod="openshift-dns/node-resolver-npnjm" Jul 27 21:03:30.345472 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.340305 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4w4r\" (UniqueName: \"kubernetes.io/projected/4c2ebad9-de50-4346-a64f-6cceecf44434-kube-api-access-s4w4r\") pod \"node-resolver-npnjm\" (UID: \"4c2ebad9-de50-4346-a64f-6cceecf44434\") " pod="openshift-dns/node-resolver-npnjm" Jul 27 21:03:30.345472 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.340474 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1907dc14-1132-4eb9-a5ee-675b43df6e6d-ovnkube-script-lib\") pod \"ovnkube-node-l6mhn\" (UID: \"1907dc14-1132-4eb9-a5ee-675b43df6e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" Jul 27 21:03:30.345472 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.340617 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1907dc14-1132-4eb9-a5ee-675b43df6e6d-host-kubelet\") pod \"ovnkube-node-l6mhn\" (UID: \"1907dc14-1132-4eb9-a5ee-675b43df6e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" Jul 27 21:03:30.345472 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.340624 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8addb952-4638-47ca-a4fb-d04fb3441aa9-cni-binary-copy\") pod \"multus-tv2n5\" (UID: \"8addb952-4638-47ca-a4fb-d04fb3441aa9\") " pod="openshift-multus/multus-tv2n5" Jul 27 21:03:30.345472 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.340707 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1907dc14-1132-4eb9-a5ee-675b43df6e6d-ovnkube-config\") pod \"ovnkube-node-l6mhn\" (UID: \"1907dc14-1132-4eb9-a5ee-675b43df6e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" Jul 27 21:03:30.345472 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.340734 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8addb952-4638-47ca-a4fb-d04fb3441aa9-host-run-netns\") pod \"multus-tv2n5\" (UID: \"8addb952-4638-47ca-a4fb-d04fb3441aa9\") " pod="openshift-multus/multus-tv2n5" Jul 27 21:03:30.345472 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.343930 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6f4b2b00-fd6c-463e-b27a-e1a925557c2f-tmp\") pod \"tuned-hq8kb\" (UID: \"6f4b2b00-fd6c-463e-b27a-e1a925557c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-hq8kb" Jul 27 21:03:30.345472 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.344173 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c2ebad9-de50-4346-a64f-6cceecf44434-tmp-dir\") pod \"node-resolver-npnjm\" (UID: \"4c2ebad9-de50-4346-a64f-6cceecf44434\") " pod="openshift-dns/node-resolver-npnjm" Jul 27 21:03:30.345472 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.344686 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1907dc14-1132-4eb9-a5ee-675b43df6e6d-ovn-node-metrics-cert\") pod \"ovnkube-node-l6mhn\" (UID: \"1907dc14-1132-4eb9-a5ee-675b43df6e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" Jul 27 21:03:30.346133 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.345913 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6f4b2b00-fd6c-463e-b27a-e1a925557c2f-etc-tuned\") pod \"tuned-hq8kb\" (UID: \"6f4b2b00-fd6c-463e-b27a-e1a925557c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-hq8kb" Jul 27 21:03:30.347233 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.346683 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/de32d9f8-5f7d-4a68-8aa4-2830c11db8f5-agent-certs\") pod \"konnectivity-agent-tx4cs\" (UID: \"de32d9f8-5f7d-4a68-8aa4-2830c11db8f5\") " pod="kube-system/konnectivity-agent-tx4cs" Jul 27 21:03:30.347233 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:30.346703 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 21:03:30.347233 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:30.346738 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 21:03:30.347233 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:30.346758 2577 projected.go:196] Error preparing data for projected volume kube-api-access-ft79h for pod openshift-network-diagnostics/network-check-target-lrnv5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 21:03:30.347233 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:30.346860 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8157bc07-d223-44de-aee6-ffa0c683a2cb-kube-api-access-ft79h podName:8157bc07-d223-44de-aee6-ffa0c683a2cb nodeName:}" failed. No retries permitted until 2026-07-27 21:03:30.846826995 +0000 UTC m=+2.978443252 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ft79h" (UniqueName: "kubernetes.io/projected/8157bc07-d223-44de-aee6-ffa0c683a2cb-kube-api-access-ft79h") pod "network-check-target-lrnv5" (UID: "8157bc07-d223-44de-aee6-ffa0c683a2cb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 21:03:30.352362 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.352257 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p2lc\" (UniqueName: \"kubernetes.io/projected/0c32154a-b41d-45c4-afa6-240a8ea24835-kube-api-access-7p2lc\") pod \"network-metrics-daemon-lvg8z\" (UID: \"0c32154a-b41d-45c4-afa6-240a8ea24835\") " pod="openshift-multus/network-metrics-daemon-lvg8z" Jul 27 21:03:30.352362 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.352329 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v82f\" (UniqueName: \"kubernetes.io/projected/f8ef5264-95a3-4193-ae7d-f0f10a7733ab-kube-api-access-5v82f\") pod \"iptables-alerter-64nks\" (UID: \"f8ef5264-95a3-4193-ae7d-f0f10a7733ab\") " pod="openshift-network-operator/iptables-alerter-64nks" Jul 27 21:03:30.352564 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.352537 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4w4r\" (UniqueName: \"kubernetes.io/projected/4c2ebad9-de50-4346-a64f-6cceecf44434-kube-api-access-s4w4r\") pod \"node-resolver-npnjm\" (UID: \"4c2ebad9-de50-4346-a64f-6cceecf44434\") " pod="openshift-dns/node-resolver-npnjm" Jul 27 21:03:30.353268 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.353185 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m52wq\" (UniqueName: \"kubernetes.io/projected/e7bf9687-782f-40c9-bb73-e8b4b2c6a763-kube-api-access-m52wq\") pod \"node-ca-zt59k\" (UID: \"e7bf9687-782f-40c9-bb73-e8b4b2c6a763\") " pod="openshift-image-registry/node-ca-zt59k" Jul 27 21:03:30.353574 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.353551 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjdsl\" (UniqueName: \"kubernetes.io/projected/1907dc14-1132-4eb9-a5ee-675b43df6e6d-kube-api-access-bjdsl\") pod \"ovnkube-node-l6mhn\" (UID: \"1907dc14-1132-4eb9-a5ee-675b43df6e6d\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" Jul 27 21:03:30.354342 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.354067 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq4kc\" (UniqueName: \"kubernetes.io/projected/8addb952-4638-47ca-a4fb-d04fb3441aa9-kube-api-access-zq4kc\") pod \"multus-tv2n5\" (UID: \"8addb952-4638-47ca-a4fb-d04fb3441aa9\") " pod="openshift-multus/multus-tv2n5" Jul 27 21:03:30.354342 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.354310 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fsh5\" (UniqueName: \"kubernetes.io/projected/6f4b2b00-fd6c-463e-b27a-e1a925557c2f-kube-api-access-8fsh5\") pod \"tuned-hq8kb\" (UID: \"6f4b2b00-fd6c-463e-b27a-e1a925557c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-hq8kb" Jul 27 21:03:30.365938 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.365872 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-201.ec2.internal" event={"ID":"bbc03a6c330f6e41568ceafa19359664","Type":"ContainerStarted","Data":"5d37e7842f73ed1ea3899e779988c4a203569b89619936fc6c61a341f41b091e"} Jul 27 21:03:30.368154 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.367893 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-201.ec2.internal" event={"ID":"4572a1a28313da7538ca6395c7dccdb4","Type":"ContainerStarted","Data":"13ced1d046807abcadc5d5b77700e1f9534044958eb9373285fa4b62c163bf9f"} Jul 27 21:03:30.441652 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.441606 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bbf44a86-6e26-4584-b851-86bcbf8d10f9-cnibin\") pod \"multus-additional-cni-plugins-5kfcg\" (UID: \"bbf44a86-6e26-4584-b851-86bcbf8d10f9\") " pod="openshift-multus/multus-additional-cni-plugins-5kfcg" Jul 27 21:03:30.441854 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.441665 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2f0a4058-4490-42ed-ae54-82e7ab83d15d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jz7ws\" (UID: \"2f0a4058-4490-42ed-ae54-82e7ab83d15d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jz7ws" Jul 27 21:03:30.441854 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.441695 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bbf44a86-6e26-4584-b851-86bcbf8d10f9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5kfcg\" (UID: \"bbf44a86-6e26-4584-b851-86bcbf8d10f9\") " pod="openshift-multus/multus-additional-cni-plugins-5kfcg" Jul 27 21:03:30.441854 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.441724 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlhxq\" (UniqueName: \"kubernetes.io/projected/bbf44a86-6e26-4584-b851-86bcbf8d10f9-kube-api-access-dlhxq\") pod \"multus-additional-cni-plugins-5kfcg\" (UID: \"bbf44a86-6e26-4584-b851-86bcbf8d10f9\") " pod="openshift-multus/multus-additional-cni-plugins-5kfcg" Jul 27 21:03:30.441854 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.441766 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2f0a4058-4490-42ed-ae54-82e7ab83d15d-etc-selinux\") pod \"aws-ebs-csi-driver-node-jz7ws\" (UID: \"2f0a4058-4490-42ed-ae54-82e7ab83d15d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jz7ws" Jul 27 21:03:30.441854 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.441800 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bbf44a86-6e26-4584-b851-86bcbf8d10f9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5kfcg\" (UID: \"bbf44a86-6e26-4584-b851-86bcbf8d10f9\") " pod="openshift-multus/multus-additional-cni-plugins-5kfcg" Jul 27 21:03:30.441854 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.441849 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2f0a4058-4490-42ed-ae54-82e7ab83d15d-registration-dir\") pod \"aws-ebs-csi-driver-node-jz7ws\" (UID: \"2f0a4058-4490-42ed-ae54-82e7ab83d15d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jz7ws" Jul 27 21:03:30.442141 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.441876 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2f0a4058-4490-42ed-ae54-82e7ab83d15d-sys-fs\") pod \"aws-ebs-csi-driver-node-jz7ws\" (UID: \"2f0a4058-4490-42ed-ae54-82e7ab83d15d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jz7ws" Jul 27 21:03:30.442141 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.441919 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2f0a4058-4490-42ed-ae54-82e7ab83d15d-socket-dir\") pod \"aws-ebs-csi-driver-node-jz7ws\" (UID: \"2f0a4058-4490-42ed-ae54-82e7ab83d15d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jz7ws" Jul 27 21:03:30.442141 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.441950 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bbf44a86-6e26-4584-b851-86bcbf8d10f9-system-cni-dir\") pod \"multus-additional-cni-plugins-5kfcg\" (UID: \"bbf44a86-6e26-4584-b851-86bcbf8d10f9\") " pod="openshift-multus/multus-additional-cni-plugins-5kfcg" Jul 27 21:03:30.442141 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.441975 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bbf44a86-6e26-4584-b851-86bcbf8d10f9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5kfcg\" (UID: \"bbf44a86-6e26-4584-b851-86bcbf8d10f9\") " pod="openshift-multus/multus-additional-cni-plugins-5kfcg" Jul 27 21:03:30.442141 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.442018 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj9pc\" (UniqueName: \"kubernetes.io/projected/2f0a4058-4490-42ed-ae54-82e7ab83d15d-kube-api-access-jj9pc\") pod \"aws-ebs-csi-driver-node-jz7ws\" (UID: \"2f0a4058-4490-42ed-ae54-82e7ab83d15d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jz7ws" Jul 27 21:03:30.442141 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.442046 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bbf44a86-6e26-4584-b851-86bcbf8d10f9-os-release\") pod \"multus-additional-cni-plugins-5kfcg\" (UID: \"bbf44a86-6e26-4584-b851-86bcbf8d10f9\") " pod="openshift-multus/multus-additional-cni-plugins-5kfcg" Jul 27 21:03:30.442141 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.442072 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bbf44a86-6e26-4584-b851-86bcbf8d10f9-cni-binary-copy\") pod \"multus-additional-cni-plugins-5kfcg\" (UID: \"bbf44a86-6e26-4584-b851-86bcbf8d10f9\") " pod="openshift-multus/multus-additional-cni-plugins-5kfcg" Jul 27 21:03:30.442141 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.442103 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2f0a4058-4490-42ed-ae54-82e7ab83d15d-device-dir\") pod \"aws-ebs-csi-driver-node-jz7ws\" (UID: \"2f0a4058-4490-42ed-ae54-82e7ab83d15d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jz7ws" Jul 27 21:03:30.442502 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.442196 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2f0a4058-4490-42ed-ae54-82e7ab83d15d-device-dir\") pod \"aws-ebs-csi-driver-node-jz7ws\" (UID: \"2f0a4058-4490-42ed-ae54-82e7ab83d15d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jz7ws" Jul 27 21:03:30.442502 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.442232 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2f0a4058-4490-42ed-ae54-82e7ab83d15d-socket-dir\") pod \"aws-ebs-csi-driver-node-jz7ws\" (UID: \"2f0a4058-4490-42ed-ae54-82e7ab83d15d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jz7ws" Jul 27 21:03:30.442502 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.442279 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bbf44a86-6e26-4584-b851-86bcbf8d10f9-os-release\") pod \"multus-additional-cni-plugins-5kfcg\" (UID: \"bbf44a86-6e26-4584-b851-86bcbf8d10f9\") " pod="openshift-multus/multus-additional-cni-plugins-5kfcg" Jul 27 21:03:30.442502 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.442316 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2f0a4058-4490-42ed-ae54-82e7ab83d15d-registration-dir\") pod \"aws-ebs-csi-driver-node-jz7ws\" (UID: \"2f0a4058-4490-42ed-ae54-82e7ab83d15d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jz7ws" Jul 27 21:03:30.442502 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.442368 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2f0a4058-4490-42ed-ae54-82e7ab83d15d-sys-fs\") pod \"aws-ebs-csi-driver-node-jz7ws\" (UID: \"2f0a4058-4490-42ed-ae54-82e7ab83d15d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jz7ws" Jul 27 21:03:30.442837 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.442524 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bbf44a86-6e26-4584-b851-86bcbf8d10f9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5kfcg\" (UID: \"bbf44a86-6e26-4584-b851-86bcbf8d10f9\") " pod="openshift-multus/multus-additional-cni-plugins-5kfcg" Jul 27 21:03:30.442837 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.442559 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bbf44a86-6e26-4584-b851-86bcbf8d10f9-system-cni-dir\") pod \"multus-additional-cni-plugins-5kfcg\" (UID: \"bbf44a86-6e26-4584-b851-86bcbf8d10f9\") " pod="openshift-multus/multus-additional-cni-plugins-5kfcg" Jul 27 21:03:30.442837 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.442644 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bbf44a86-6e26-4584-b851-86bcbf8d10f9-cnibin\") pod \"multus-additional-cni-plugins-5kfcg\" (UID: \"bbf44a86-6e26-4584-b851-86bcbf8d10f9\") " pod="openshift-multus/multus-additional-cni-plugins-5kfcg" Jul 27 21:03:30.442837 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.442651 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bbf44a86-6e26-4584-b851-86bcbf8d10f9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5kfcg\" (UID: \"bbf44a86-6e26-4584-b851-86bcbf8d10f9\") " pod="openshift-multus/multus-additional-cni-plugins-5kfcg" Jul 27 21:03:30.442837 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.442703 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2f0a4058-4490-42ed-ae54-82e7ab83d15d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jz7ws\" (UID: \"2f0a4058-4490-42ed-ae54-82e7ab83d15d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jz7ws" Jul 27 21:03:30.442837 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.442730 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bbf44a86-6e26-4584-b851-86bcbf8d10f9-cni-binary-copy\") pod \"multus-additional-cni-plugins-5kfcg\" (UID: \"bbf44a86-6e26-4584-b851-86bcbf8d10f9\") " pod="openshift-multus/multus-additional-cni-plugins-5kfcg" Jul 27 21:03:30.443053 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.442851 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2f0a4058-4490-42ed-ae54-82e7ab83d15d-etc-selinux\") pod \"aws-ebs-csi-driver-node-jz7ws\" (UID: \"2f0a4058-4490-42ed-ae54-82e7ab83d15d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jz7ws" Jul 27 21:03:30.443053 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.442980 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bbf44a86-6e26-4584-b851-86bcbf8d10f9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5kfcg\" (UID: \"bbf44a86-6e26-4584-b851-86bcbf8d10f9\") " pod="openshift-multus/multus-additional-cni-plugins-5kfcg" Jul 27 21:03:30.454600 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.454478 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlhxq\" (UniqueName: \"kubernetes.io/projected/bbf44a86-6e26-4584-b851-86bcbf8d10f9-kube-api-access-dlhxq\") pod \"multus-additional-cni-plugins-5kfcg\" (UID: \"bbf44a86-6e26-4584-b851-86bcbf8d10f9\") " pod="openshift-multus/multus-additional-cni-plugins-5kfcg" Jul 27 21:03:30.454827 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.454791 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj9pc\" (UniqueName: \"kubernetes.io/projected/2f0a4058-4490-42ed-ae54-82e7ab83d15d-kube-api-access-jj9pc\") pod \"aws-ebs-csi-driver-node-jz7ws\" (UID: \"2f0a4058-4490-42ed-ae54-82e7ab83d15d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jz7ws" Jul 27 21:03:30.526734 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.526696 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tx4cs" Jul 27 21:03:30.538713 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.538439 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-64nks" Jul 27 21:03:30.538925 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:30.538889 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde32d9f8_5f7d_4a68_8aa4_2830c11db8f5.slice/crio-d364254259b4eabbb9ef4ee05da8e5abedc6cb61bbf723592403fab077ac0589 WatchSource:0}: Error finding container d364254259b4eabbb9ef4ee05da8e5abedc6cb61bbf723592403fab077ac0589: Status 404 returned error can't find the container with id d364254259b4eabbb9ef4ee05da8e5abedc6cb61bbf723592403fab077ac0589 Jul 27 21:03:30.551059 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.551030 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" Jul 27 21:03:30.557308 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.557277 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hq8kb" Jul 27 21:03:30.565303 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.565272 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-npnjm" Jul 27 21:03:30.573071 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.573037 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tv2n5" Jul 27 21:03:30.579961 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.579932 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zt59k" Jul 27 21:03:30.587302 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.587270 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jz7ws" Jul 27 21:03:30.595257 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.595225 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5kfcg" Jul 27 21:03:30.846289 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.846191 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c77555ba-462e-49c5-b8b0-e83987e8a2e3-original-pull-secret\") pod \"global-pull-secret-syncer-l2czx\" (UID: \"c77555ba-462e-49c5-b8b0-e83987e8a2e3\") " pod="kube-system/global-pull-secret-syncer-l2czx" Jul 27 21:03:30.846289 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.846245 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c32154a-b41d-45c4-afa6-240a8ea24835-metrics-certs\") pod \"network-metrics-daemon-lvg8z\" (UID: \"0c32154a-b41d-45c4-afa6-240a8ea24835\") " pod="openshift-multus/network-metrics-daemon-lvg8z" Jul 27 21:03:30.846511 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:30.846421 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 21:03:30.846577 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:30.846518 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c77555ba-462e-49c5-b8b0-e83987e8a2e3-original-pull-secret podName:c77555ba-462e-49c5-b8b0-e83987e8a2e3 nodeName:}" failed. No retries permitted until 2026-07-27 21:03:31.846491315 +0000 UTC m=+3.978107529 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c77555ba-462e-49c5-b8b0-e83987e8a2e3-original-pull-secret") pod "global-pull-secret-syncer-l2czx" (UID: "c77555ba-462e-49c5-b8b0-e83987e8a2e3") : object "kube-system"/"original-pull-secret" not registered Jul 27 21:03:30.846577 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:30.846530 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 21:03:30.846714 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:30.846590 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c32154a-b41d-45c4-afa6-240a8ea24835-metrics-certs podName:0c32154a-b41d-45c4-afa6-240a8ea24835 nodeName:}" failed. No retries permitted until 2026-07-27 21:03:31.846571173 +0000 UTC m=+3.978187400 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0c32154a-b41d-45c4-afa6-240a8ea24835-metrics-certs") pod "network-metrics-daemon-lvg8z" (UID: "0c32154a-b41d-45c4-afa6-240a8ea24835") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 21:03:30.885958 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:30.885917 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8ef5264_95a3_4193_ae7d_f0f10a7733ab.slice/crio-5c4f8d90dc92c98e5edd10b2261d68bbf2fedfec8c285a9f29492b4b4214239d WatchSource:0}: Error finding container 5c4f8d90dc92c98e5edd10b2261d68bbf2fedfec8c285a9f29492b4b4214239d: Status 404 returned error can't find the container with id 5c4f8d90dc92c98e5edd10b2261d68bbf2fedfec8c285a9f29492b4b4214239d Jul 27 21:03:30.947675 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:30.947632 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft79h\" (UniqueName: \"kubernetes.io/projected/8157bc07-d223-44de-aee6-ffa0c683a2cb-kube-api-access-ft79h\") pod \"network-check-target-lrnv5\" (UID: \"8157bc07-d223-44de-aee6-ffa0c683a2cb\") " pod="openshift-network-diagnostics/network-check-target-lrnv5" Jul 27 21:03:30.947874 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:30.947822 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 21:03:30.947874 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:30.947854 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 21:03:30.947874 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:30.947867 2577 projected.go:196] Error preparing data for projected volume kube-api-access-ft79h for pod openshift-network-diagnostics/network-check-target-lrnv5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 21:03:30.948023 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:30.947936 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8157bc07-d223-44de-aee6-ffa0c683a2cb-kube-api-access-ft79h podName:8157bc07-d223-44de-aee6-ffa0c683a2cb nodeName:}" failed. No retries permitted until 2026-07-27 21:03:31.947914709 +0000 UTC m=+4.079530916 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ft79h" (UniqueName: "kubernetes.io/projected/8157bc07-d223-44de-aee6-ffa0c683a2cb-kube-api-access-ft79h") pod "network-check-target-lrnv5" (UID: "8157bc07-d223-44de-aee6-ffa0c683a2cb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 21:03:31.271364 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:31.271258 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 20:58:30 +0000 UTC" deadline="2028-05-02 22:38:21.437011912 +0000 UTC" Jul 27 21:03:31.271364 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:31.271296 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15481h34m50.16571996s" Jul 27 21:03:31.360639 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:31.360601 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lrnv5" Jul 27 21:03:31.360822 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:31.360736 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lrnv5" podUID="8157bc07-d223-44de-aee6-ffa0c683a2cb" Jul 27 21:03:31.360822 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:31.360801 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lvg8z" Jul 27 21:03:31.360967 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:31.360940 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lvg8z" podUID="0c32154a-b41d-45c4-afa6-240a8ea24835" Jul 27 21:03:31.371412 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:31.371371 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-64nks" event={"ID":"f8ef5264-95a3-4193-ae7d-f0f10a7733ab","Type":"ContainerStarted","Data":"5c4f8d90dc92c98e5edd10b2261d68bbf2fedfec8c285a9f29492b4b4214239d"} Jul 27 21:03:31.372705 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:31.372671 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tx4cs" event={"ID":"de32d9f8-5f7d-4a68-8aa4-2830c11db8f5","Type":"ContainerStarted","Data":"d364254259b4eabbb9ef4ee05da8e5abedc6cb61bbf723592403fab077ac0589"} Jul 27 21:03:31.640980 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:31.640925 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f0a4058_4490_42ed_ae54_82e7ab83d15d.slice/crio-1eae1607c8eaacfedf705c289106d670d9703683484ec19a5c72537b9179f78d WatchSource:0}: Error finding container 1eae1607c8eaacfedf705c289106d670d9703683484ec19a5c72537b9179f78d: Status 404 returned error can't find the container with id 1eae1607c8eaacfedf705c289106d670d9703683484ec19a5c72537b9179f78d Jul 27 21:03:31.648015 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:31.647981 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c2ebad9_de50_4346_a64f_6cceecf44434.slice/crio-642f3217391c1fc631b760a8ff8429fabf843377e75c3f27337c111b26f06a90 WatchSource:0}: Error finding container 642f3217391c1fc631b760a8ff8429fabf843377e75c3f27337c111b26f06a90: Status 404 returned error can't find the container with id 642f3217391c1fc631b760a8ff8429fabf843377e75c3f27337c111b26f06a90 Jul 27 21:03:31.648778 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:31.648744 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbf44a86_6e26_4584_b851_86bcbf8d10f9.slice/crio-761a95715d3f5f4b17d3517dfe32b281f467b8e9b57ebcff112f7a8e90a36c9d WatchSource:0}: Error finding container 761a95715d3f5f4b17d3517dfe32b281f467b8e9b57ebcff112f7a8e90a36c9d: Status 404 returned error can't find the container with id 761a95715d3f5f4b17d3517dfe32b281f467b8e9b57ebcff112f7a8e90a36c9d Jul 27 21:03:31.651009 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:31.650980 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8addb952_4638_47ca_a4fb_d04fb3441aa9.slice/crio-405115ff9741b8168be937aac01b67a2098f70a6daa494ed35ccc9ecf45b7235 WatchSource:0}: Error finding container 405115ff9741b8168be937aac01b67a2098f70a6daa494ed35ccc9ecf45b7235: Status 404 returned error can't find the container with id 405115ff9741b8168be937aac01b67a2098f70a6daa494ed35ccc9ecf45b7235 Jul 27 21:03:31.654584 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:31.653907 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f4b2b00_fd6c_463e_b27a_e1a925557c2f.slice/crio-96dd8288aa060025250e925a5a77a1f4dc75d83d656495aa2ae0038fb3cc2af5 WatchSource:0}: Error finding container 96dd8288aa060025250e925a5a77a1f4dc75d83d656495aa2ae0038fb3cc2af5: Status 404 returned error can't find the container with id 96dd8288aa060025250e925a5a77a1f4dc75d83d656495aa2ae0038fb3cc2af5 Jul 27 21:03:31.655040 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:31.655013 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7bf9687_782f_40c9_bb73_e8b4b2c6a763.slice/crio-05f6e67adcb59a85574d6f60c87d7f8dfec98bbe9cdc1f5d58deb16b80fb8ee6 WatchSource:0}: Error finding container 05f6e67adcb59a85574d6f60c87d7f8dfec98bbe9cdc1f5d58deb16b80fb8ee6: Status 404 returned error can't find the container with id 05f6e67adcb59a85574d6f60c87d7f8dfec98bbe9cdc1f5d58deb16b80fb8ee6 Jul 27 21:03:31.656484 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:03:31.656378 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1907dc14_1132_4eb9_a5ee_675b43df6e6d.slice/crio-82b74c6b1f2bd07e616ea8ad026439155616317745075202c8441860faafb5de WatchSource:0}: Error finding container 82b74c6b1f2bd07e616ea8ad026439155616317745075202c8441860faafb5de: Status 404 returned error can't find the container with id 82b74c6b1f2bd07e616ea8ad026439155616317745075202c8441860faafb5de Jul 27 21:03:31.854682 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:31.854380 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c77555ba-462e-49c5-b8b0-e83987e8a2e3-original-pull-secret\") pod \"global-pull-secret-syncer-l2czx\" (UID: \"c77555ba-462e-49c5-b8b0-e83987e8a2e3\") " pod="kube-system/global-pull-secret-syncer-l2czx" Jul 27 21:03:31.855054 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:31.854725 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c32154a-b41d-45c4-afa6-240a8ea24835-metrics-certs\") pod \"network-metrics-daemon-lvg8z\" (UID: \"0c32154a-b41d-45c4-afa6-240a8ea24835\") " pod="openshift-multus/network-metrics-daemon-lvg8z" Jul 27 21:03:31.855054 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:31.854562 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 21:03:31.855054 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:31.854821 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c77555ba-462e-49c5-b8b0-e83987e8a2e3-original-pull-secret podName:c77555ba-462e-49c5-b8b0-e83987e8a2e3 nodeName:}" failed. No retries permitted until 2026-07-27 21:03:33.854796623 +0000 UTC m=+5.986412842 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c77555ba-462e-49c5-b8b0-e83987e8a2e3-original-pull-secret") pod "global-pull-secret-syncer-l2czx" (UID: "c77555ba-462e-49c5-b8b0-e83987e8a2e3") : object "kube-system"/"original-pull-secret" not registered Jul 27 21:03:31.855054 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:31.854874 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 21:03:31.855054 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:31.854927 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c32154a-b41d-45c4-afa6-240a8ea24835-metrics-certs podName:0c32154a-b41d-45c4-afa6-240a8ea24835 nodeName:}" failed. No retries permitted until 2026-07-27 21:03:33.854911547 +0000 UTC m=+5.986527758 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0c32154a-b41d-45c4-afa6-240a8ea24835-metrics-certs") pod "network-metrics-daemon-lvg8z" (UID: "0c32154a-b41d-45c4-afa6-240a8ea24835") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 21:03:31.955426 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:31.955073 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft79h\" (UniqueName: \"kubernetes.io/projected/8157bc07-d223-44de-aee6-ffa0c683a2cb-kube-api-access-ft79h\") pod \"network-check-target-lrnv5\" (UID: \"8157bc07-d223-44de-aee6-ffa0c683a2cb\") " pod="openshift-network-diagnostics/network-check-target-lrnv5" Jul 27 21:03:31.955426 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:31.955211 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 21:03:31.955426 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:31.955226 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 21:03:31.955426 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:31.955238 2577 projected.go:196] Error preparing data for projected volume kube-api-access-ft79h for pod openshift-network-diagnostics/network-check-target-lrnv5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 21:03:31.955426 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:31.955286 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8157bc07-d223-44de-aee6-ffa0c683a2cb-kube-api-access-ft79h podName:8157bc07-d223-44de-aee6-ffa0c683a2cb nodeName:}" failed. No retries permitted until 2026-07-27 21:03:33.955272927 +0000 UTC m=+6.086889144 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ft79h" (UniqueName: "kubernetes.io/projected/8157bc07-d223-44de-aee6-ffa0c683a2cb-kube-api-access-ft79h") pod "network-check-target-lrnv5" (UID: "8157bc07-d223-44de-aee6-ffa0c683a2cb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 21:03:32.272151 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:32.271994 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 20:58:30 +0000 UTC" deadline="2028-05-07 12:51:43.707977468 +0000 UTC" Jul 27 21:03:32.272151 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:32.272064 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15591h48m11.435918931s" Jul 27 21:03:32.362061 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:32.361375 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2czx" Jul 27 21:03:32.362061 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:32.361531 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2czx" podUID="c77555ba-462e-49c5-b8b0-e83987e8a2e3" Jul 27 21:03:32.377702 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:32.377667 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-201.ec2.internal" event={"ID":"4572a1a28313da7538ca6395c7dccdb4","Type":"ContainerStarted","Data":"53c3f3818105537b89828f86a04c579cb653985994e84086a7420efdf20e3950"} Jul 27 21:03:32.379293 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:32.379249 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" event={"ID":"1907dc14-1132-4eb9-a5ee-675b43df6e6d","Type":"ContainerStarted","Data":"82b74c6b1f2bd07e616ea8ad026439155616317745075202c8441860faafb5de"} Jul 27 21:03:32.380765 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:32.380735 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zt59k" event={"ID":"e7bf9687-782f-40c9-bb73-e8b4b2c6a763","Type":"ContainerStarted","Data":"05f6e67adcb59a85574d6f60c87d7f8dfec98bbe9cdc1f5d58deb16b80fb8ee6"} Jul 27 21:03:32.382626 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:32.382592 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-npnjm" event={"ID":"4c2ebad9-de50-4346-a64f-6cceecf44434","Type":"ContainerStarted","Data":"642f3217391c1fc631b760a8ff8429fabf843377e75c3f27337c111b26f06a90"} Jul 27 21:03:32.383960 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:32.383932 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jz7ws" event={"ID":"2f0a4058-4490-42ed-ae54-82e7ab83d15d","Type":"ContainerStarted","Data":"1eae1607c8eaacfedf705c289106d670d9703683484ec19a5c72537b9179f78d"} Jul 27 21:03:32.385685 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:32.385658 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-201.ec2.internal" event={"ID":"bbc03a6c330f6e41568ceafa19359664","Type":"ContainerStarted","Data":"91eb3e244aae5727f628432081c8ede6f8b04aa76623cb2a04d3d751dc1cd271"} Jul 27 21:03:32.387057 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:32.387033 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hq8kb" event={"ID":"6f4b2b00-fd6c-463e-b27a-e1a925557c2f","Type":"ContainerStarted","Data":"96dd8288aa060025250e925a5a77a1f4dc75d83d656495aa2ae0038fb3cc2af5"} Jul 27 21:03:32.388261 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:32.388238 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tv2n5" event={"ID":"8addb952-4638-47ca-a4fb-d04fb3441aa9","Type":"ContainerStarted","Data":"405115ff9741b8168be937aac01b67a2098f70a6daa494ed35ccc9ecf45b7235"} Jul 27 21:03:32.389449 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:32.389421 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5kfcg" event={"ID":"bbf44a86-6e26-4584-b851-86bcbf8d10f9","Type":"ContainerStarted","Data":"761a95715d3f5f4b17d3517dfe32b281f467b8e9b57ebcff112f7a8e90a36c9d"} Jul 27 21:03:32.393119 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:32.393066 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-201.ec2.internal" podStartSLOduration=2.39304612 podStartE2EDuration="2.39304612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:03:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:03:32.392482615 +0000 UTC m=+4.524098843" watchObservedRunningTime="2026-07-27 21:03:32.39304612 +0000 UTC m=+4.524662352" Jul 27 21:03:33.360805 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:33.360695 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lrnv5" Jul 27 21:03:33.361317 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:33.360857 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lrnv5" podUID="8157bc07-d223-44de-aee6-ffa0c683a2cb" Jul 27 21:03:33.361385 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:33.361330 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lvg8z" Jul 27 21:03:33.361503 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:33.361471 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lvg8z" podUID="0c32154a-b41d-45c4-afa6-240a8ea24835" Jul 27 21:03:33.416302 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:33.414680 2577 generic.go:356] "Generic (PLEG): container finished" podID="bbc03a6c330f6e41568ceafa19359664" containerID="91eb3e244aae5727f628432081c8ede6f8b04aa76623cb2a04d3d751dc1cd271" exitCode=0 Jul 27 21:03:33.416302 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:33.414803 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-201.ec2.internal" event={"ID":"bbc03a6c330f6e41568ceafa19359664","Type":"ContainerDied","Data":"91eb3e244aae5727f628432081c8ede6f8b04aa76623cb2a04d3d751dc1cd271"} Jul 27 21:03:33.422885 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:33.422836 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tx4cs" event={"ID":"de32d9f8-5f7d-4a68-8aa4-2830c11db8f5","Type":"ContainerStarted","Data":"239c4876fae65e2db57ff37c16d4fb9d1faede32e46c031253e47974ad9bebd2"} Jul 27 21:03:33.450302 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:33.450200 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-tx4cs" podStartSLOduration=2.500703585 podStartE2EDuration="4.45017333s" totalImagesPullingTime="1.949469745s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:03:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:03:33.449664806 +0000 UTC m=+5.581281027" watchObservedRunningTime="2026-07-27 21:03:33.45017333 +0000 UTC m=+5.581789559" Jul 27 21:03:33.874049 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:33.873955 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c77555ba-462e-49c5-b8b0-e83987e8a2e3-original-pull-secret\") pod \"global-pull-secret-syncer-l2czx\" (UID: \"c77555ba-462e-49c5-b8b0-e83987e8a2e3\") " pod="kube-system/global-pull-secret-syncer-l2czx" Jul 27 21:03:33.874049 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:33.874015 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c32154a-b41d-45c4-afa6-240a8ea24835-metrics-certs\") pod \"network-metrics-daemon-lvg8z\" (UID: \"0c32154a-b41d-45c4-afa6-240a8ea24835\") " pod="openshift-multus/network-metrics-daemon-lvg8z" Jul 27 21:03:33.874294 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:33.874203 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 21:03:33.874294 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:33.874267 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c32154a-b41d-45c4-afa6-240a8ea24835-metrics-certs podName:0c32154a-b41d-45c4-afa6-240a8ea24835 nodeName:}" failed. No retries permitted until 2026-07-27 21:03:37.874248064 +0000 UTC m=+10.005864270 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0c32154a-b41d-45c4-afa6-240a8ea24835-metrics-certs") pod "network-metrics-daemon-lvg8z" (UID: "0c32154a-b41d-45c4-afa6-240a8ea24835") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 21:03:33.874730 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:33.874706 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 21:03:33.874834 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:33.874766 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c77555ba-462e-49c5-b8b0-e83987e8a2e3-original-pull-secret podName:c77555ba-462e-49c5-b8b0-e83987e8a2e3 nodeName:}" failed. No retries permitted until 2026-07-27 21:03:37.874750639 +0000 UTC m=+10.006366846 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c77555ba-462e-49c5-b8b0-e83987e8a2e3-original-pull-secret") pod "global-pull-secret-syncer-l2czx" (UID: "c77555ba-462e-49c5-b8b0-e83987e8a2e3") : object "kube-system"/"original-pull-secret" not registered Jul 27 21:03:33.975216 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:33.975140 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft79h\" (UniqueName: \"kubernetes.io/projected/8157bc07-d223-44de-aee6-ffa0c683a2cb-kube-api-access-ft79h\") pod \"network-check-target-lrnv5\" (UID: \"8157bc07-d223-44de-aee6-ffa0c683a2cb\") " pod="openshift-network-diagnostics/network-check-target-lrnv5" Jul 27 21:03:33.975434 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:33.975406 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 21:03:33.975513 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:33.975442 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 21:03:33.975513 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:33.975459 2577 projected.go:196] Error preparing data for projected volume kube-api-access-ft79h for pod openshift-network-diagnostics/network-check-target-lrnv5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 21:03:33.975672 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:33.975527 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8157bc07-d223-44de-aee6-ffa0c683a2cb-kube-api-access-ft79h podName:8157bc07-d223-44de-aee6-ffa0c683a2cb nodeName:}" failed. No retries permitted until 2026-07-27 21:03:37.975506533 +0000 UTC m=+10.107122748 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ft79h" (UniqueName: "kubernetes.io/projected/8157bc07-d223-44de-aee6-ffa0c683a2cb-kube-api-access-ft79h") pod "network-check-target-lrnv5" (UID: "8157bc07-d223-44de-aee6-ffa0c683a2cb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 21:03:34.361841 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:34.361741 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2czx" Jul 27 21:03:34.362301 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:34.361886 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2czx" podUID="c77555ba-462e-49c5-b8b0-e83987e8a2e3" Jul 27 21:03:35.360796 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:35.360758 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lvg8z" Jul 27 21:03:35.361004 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:35.360817 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lrnv5" Jul 27 21:03:35.361004 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:35.360926 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lvg8z" podUID="0c32154a-b41d-45c4-afa6-240a8ea24835" Jul 27 21:03:35.361435 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:35.361384 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lrnv5" podUID="8157bc07-d223-44de-aee6-ffa0c683a2cb" Jul 27 21:03:36.361780 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:36.361745 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2czx" Jul 27 21:03:36.362272 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:36.361896 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2czx" podUID="c77555ba-462e-49c5-b8b0-e83987e8a2e3" Jul 27 21:03:37.360859 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:37.360762 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lrnv5" Jul 27 21:03:37.361036 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:37.360762 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lvg8z" Jul 27 21:03:37.361036 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:37.360956 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lrnv5" podUID="8157bc07-d223-44de-aee6-ffa0c683a2cb" Jul 27 21:03:37.361036 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:37.360974 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lvg8z" podUID="0c32154a-b41d-45c4-afa6-240a8ea24835" Jul 27 21:03:37.911073 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:37.911026 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c77555ba-462e-49c5-b8b0-e83987e8a2e3-original-pull-secret\") pod \"global-pull-secret-syncer-l2czx\" (UID: \"c77555ba-462e-49c5-b8b0-e83987e8a2e3\") " pod="kube-system/global-pull-secret-syncer-l2czx" Jul 27 21:03:37.911594 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:37.911089 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c32154a-b41d-45c4-afa6-240a8ea24835-metrics-certs\") pod \"network-metrics-daemon-lvg8z\" (UID: \"0c32154a-b41d-45c4-afa6-240a8ea24835\") " pod="openshift-multus/network-metrics-daemon-lvg8z" Jul 27 21:03:37.911594 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:37.911175 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 21:03:37.911594 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:37.911225 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 21:03:37.911594 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:37.911245 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c77555ba-462e-49c5-b8b0-e83987e8a2e3-original-pull-secret podName:c77555ba-462e-49c5-b8b0-e83987e8a2e3 nodeName:}" failed. No retries permitted until 2026-07-27 21:03:45.911226136 +0000 UTC m=+18.042842358 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c77555ba-462e-49c5-b8b0-e83987e8a2e3-original-pull-secret") pod "global-pull-secret-syncer-l2czx" (UID: "c77555ba-462e-49c5-b8b0-e83987e8a2e3") : object "kube-system"/"original-pull-secret" not registered Jul 27 21:03:37.911594 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:37.911263 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c32154a-b41d-45c4-afa6-240a8ea24835-metrics-certs podName:0c32154a-b41d-45c4-afa6-240a8ea24835 nodeName:}" failed. No retries permitted until 2026-07-27 21:03:45.911251953 +0000 UTC m=+18.042868178 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0c32154a-b41d-45c4-afa6-240a8ea24835-metrics-certs") pod "network-metrics-daemon-lvg8z" (UID: "0c32154a-b41d-45c4-afa6-240a8ea24835") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 21:03:38.012222 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:38.011990 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft79h\" (UniqueName: \"kubernetes.io/projected/8157bc07-d223-44de-aee6-ffa0c683a2cb-kube-api-access-ft79h\") pod \"network-check-target-lrnv5\" (UID: \"8157bc07-d223-44de-aee6-ffa0c683a2cb\") " pod="openshift-network-diagnostics/network-check-target-lrnv5" Jul 27 21:03:38.012401 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:38.012265 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 21:03:38.012401 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:38.012288 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 21:03:38.012401 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:38.012301 2577 projected.go:196] Error preparing data for projected volume kube-api-access-ft79h for pod openshift-network-diagnostics/network-check-target-lrnv5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 21:03:38.012401 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:38.012364 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8157bc07-d223-44de-aee6-ffa0c683a2cb-kube-api-access-ft79h podName:8157bc07-d223-44de-aee6-ffa0c683a2cb nodeName:}" failed. No retries permitted until 2026-07-27 21:03:46.012346526 +0000 UTC m=+18.143962728 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ft79h" (UniqueName: "kubernetes.io/projected/8157bc07-d223-44de-aee6-ffa0c683a2cb-kube-api-access-ft79h") pod "network-check-target-lrnv5" (UID: "8157bc07-d223-44de-aee6-ffa0c683a2cb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 21:03:38.102143 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:38.102105 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-tx4cs" Jul 27 21:03:38.102975 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:38.102952 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-tx4cs" Jul 27 21:03:38.362053 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:38.361970 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2czx" Jul 27 21:03:38.362255 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:38.362106 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2czx" podUID="c77555ba-462e-49c5-b8b0-e83987e8a2e3" Jul 27 21:03:38.438490 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:38.438452 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-tx4cs" Jul 27 21:03:38.439232 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:38.439201 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-tx4cs" Jul 27 21:03:39.361410 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:39.361375 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lrnv5" Jul 27 21:03:39.361887 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:39.361516 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lrnv5" podUID="8157bc07-d223-44de-aee6-ffa0c683a2cb" Jul 27 21:03:39.361887 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:39.361811 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lvg8z" Jul 27 21:03:39.362016 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:39.361921 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lvg8z" podUID="0c32154a-b41d-45c4-afa6-240a8ea24835" Jul 27 21:03:40.360937 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:40.360901 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2czx" Jul 27 21:03:40.361120 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:40.361043 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2czx" podUID="c77555ba-462e-49c5-b8b0-e83987e8a2e3" Jul 27 21:03:41.360341 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:41.360301 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lrnv5" Jul 27 21:03:41.360900 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:41.360318 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lvg8z" Jul 27 21:03:41.360900 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:41.360421 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lrnv5" podUID="8157bc07-d223-44de-aee6-ffa0c683a2cb" Jul 27 21:03:41.360900 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:41.360511 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lvg8z" podUID="0c32154a-b41d-45c4-afa6-240a8ea24835" Jul 27 21:03:42.360748 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:42.360702 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2czx" Jul 27 21:03:42.361204 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:42.360849 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2czx" podUID="c77555ba-462e-49c5-b8b0-e83987e8a2e3" Jul 27 21:03:43.361046 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:43.361005 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lrnv5" Jul 27 21:03:43.361442 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:43.361143 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lrnv5" podUID="8157bc07-d223-44de-aee6-ffa0c683a2cb" Jul 27 21:03:43.361442 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:43.361212 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lvg8z" Jul 27 21:03:43.361442 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:43.361349 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lvg8z" podUID="0c32154a-b41d-45c4-afa6-240a8ea24835" Jul 27 21:03:44.360642 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:44.360602 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2czx" Jul 27 21:03:44.360813 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:44.360749 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2czx" podUID="c77555ba-462e-49c5-b8b0-e83987e8a2e3" Jul 27 21:03:45.360827 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:45.360785 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lrnv5" Jul 27 21:03:45.361342 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:45.360785 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lvg8z" Jul 27 21:03:45.361342 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:45.360932 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lrnv5" podUID="8157bc07-d223-44de-aee6-ffa0c683a2cb" Jul 27 21:03:45.361342 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:45.360979 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lvg8z" podUID="0c32154a-b41d-45c4-afa6-240a8ea24835" Jul 27 21:03:45.976242 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:45.976203 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c77555ba-462e-49c5-b8b0-e83987e8a2e3-original-pull-secret\") pod \"global-pull-secret-syncer-l2czx\" (UID: \"c77555ba-462e-49c5-b8b0-e83987e8a2e3\") " pod="kube-system/global-pull-secret-syncer-l2czx" Jul 27 21:03:45.976242 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:45.976244 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c32154a-b41d-45c4-afa6-240a8ea24835-metrics-certs\") pod \"network-metrics-daemon-lvg8z\" (UID: \"0c32154a-b41d-45c4-afa6-240a8ea24835\") " pod="openshift-multus/network-metrics-daemon-lvg8z" Jul 27 21:03:45.976442 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:45.976347 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 21:03:45.976442 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:45.976362 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 21:03:45.976442 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:45.976403 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c32154a-b41d-45c4-afa6-240a8ea24835-metrics-certs podName:0c32154a-b41d-45c4-afa6-240a8ea24835 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:01.976388185 +0000 UTC m=+34.108004386 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0c32154a-b41d-45c4-afa6-240a8ea24835-metrics-certs") pod "network-metrics-daemon-lvg8z" (UID: "0c32154a-b41d-45c4-afa6-240a8ea24835") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 21:03:45.976442 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:45.976416 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c77555ba-462e-49c5-b8b0-e83987e8a2e3-original-pull-secret podName:c77555ba-462e-49c5-b8b0-e83987e8a2e3 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:01.976410198 +0000 UTC m=+34.108026399 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c77555ba-462e-49c5-b8b0-e83987e8a2e3-original-pull-secret") pod "global-pull-secret-syncer-l2czx" (UID: "c77555ba-462e-49c5-b8b0-e83987e8a2e3") : object "kube-system"/"original-pull-secret" not registered Jul 27 21:03:46.077511 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:46.077417 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft79h\" (UniqueName: \"kubernetes.io/projected/8157bc07-d223-44de-aee6-ffa0c683a2cb-kube-api-access-ft79h\") pod \"network-check-target-lrnv5\" (UID: \"8157bc07-d223-44de-aee6-ffa0c683a2cb\") " pod="openshift-network-diagnostics/network-check-target-lrnv5" Jul 27 21:03:46.077706 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:46.077599 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 21:03:46.077706 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:46.077626 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 21:03:46.077706 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:46.077638 2577 projected.go:196] Error preparing data for projected volume kube-api-access-ft79h for pod openshift-network-diagnostics/network-check-target-lrnv5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 21:03:46.077706 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:46.077697 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8157bc07-d223-44de-aee6-ffa0c683a2cb-kube-api-access-ft79h podName:8157bc07-d223-44de-aee6-ffa0c683a2cb nodeName:}" failed. No retries permitted until 2026-07-27 21:04:02.077681 +0000 UTC m=+34.209297202 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ft79h" (UniqueName: "kubernetes.io/projected/8157bc07-d223-44de-aee6-ffa0c683a2cb-kube-api-access-ft79h") pod "network-check-target-lrnv5" (UID: "8157bc07-d223-44de-aee6-ffa0c683a2cb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 21:03:46.361002 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:46.360917 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2czx" Jul 27 21:03:46.361439 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:46.361057 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2czx" podUID="c77555ba-462e-49c5-b8b0-e83987e8a2e3" Jul 27 21:03:47.361212 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:47.361173 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lrnv5" Jul 27 21:03:47.361705 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:47.361173 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lvg8z" Jul 27 21:03:47.361705 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:47.361297 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lrnv5" podUID="8157bc07-d223-44de-aee6-ffa0c683a2cb" Jul 27 21:03:47.361705 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:47.361361 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lvg8z" podUID="0c32154a-b41d-45c4-afa6-240a8ea24835" Jul 27 21:03:48.362031 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:48.361985 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2czx" Jul 27 21:03:48.362519 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:48.362119 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2czx" podUID="c77555ba-462e-49c5-b8b0-e83987e8a2e3" Jul 27 21:03:49.361101 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:49.361055 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lrnv5" Jul 27 21:03:49.361270 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:49.361184 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lrnv5" podUID="8157bc07-d223-44de-aee6-ffa0c683a2cb" Jul 27 21:03:49.361270 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:49.361237 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lvg8z" Jul 27 21:03:49.361407 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:49.361360 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lvg8z" podUID="0c32154a-b41d-45c4-afa6-240a8ea24835" Jul 27 21:03:50.361639 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:50.360366 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2czx" Jul 27 21:03:50.361639 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:50.360713 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2czx" podUID="c77555ba-462e-49c5-b8b0-e83987e8a2e3" Jul 27 21:03:50.462176 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:50.462135 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zt59k" event={"ID":"e7bf9687-782f-40c9-bb73-e8b4b2c6a763","Type":"ContainerStarted","Data":"185f07baebeed0266108643ab5f41680ec8f8f81923a3ee60ad34fa9ca89cd7d"} Jul 27 21:03:50.464119 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:50.464076 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-npnjm" event={"ID":"4c2ebad9-de50-4346-a64f-6cceecf44434","Type":"ContainerStarted","Data":"14d7321c22b3d3e8674689e83577c5c58b4caed6d9ec1d81c30e2efba35aaa09"} Jul 27 21:03:50.465690 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:50.465661 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jz7ws" event={"ID":"2f0a4058-4490-42ed-ae54-82e7ab83d15d","Type":"ContainerStarted","Data":"5ad2c42b0f1ed5f877122bbf4123ecac83ec22b4223049122f2b6a1979e1d859"} Jul 27 21:03:50.467813 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:50.467784 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-201.ec2.internal" event={"ID":"bbc03a6c330f6e41568ceafa19359664","Type":"ContainerStarted","Data":"185e4ee7ff2d369c516be4a069c11b685f88740a260c5599a02b470a3fa0041b"} Jul 27 21:03:50.469388 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:50.469360 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hq8kb" event={"ID":"6f4b2b00-fd6c-463e-b27a-e1a925557c2f","Type":"ContainerStarted","Data":"dfb414374daa14977f41d061644747e52aa17d990063ee36bfafe5127e358640"} Jul 27 21:03:50.470977 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:50.470940 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tv2n5" event={"ID":"8addb952-4638-47ca-a4fb-d04fb3441aa9","Type":"ContainerStarted","Data":"40df6b1756c58c7a17ac03c8a2db440338b06571996aec9cce48e3f1868cd6ac"} Jul 27 21:03:50.472447 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:50.472420 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5kfcg" event={"ID":"bbf44a86-6e26-4584-b851-86bcbf8d10f9","Type":"ContainerStarted","Data":"a398c6192ef7d4cbbd85206dec9f3ce0d2b9bd5f5f81519df7f63bf4cd09ff8d"} Jul 27 21:03:50.478474 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:50.478429 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-zt59k" podStartSLOduration=2.946658539 podStartE2EDuration="21.478413682s" totalImagesPullingTime="18.531755143s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:03:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:03:50.477925892 +0000 UTC m=+22.609542137" watchObservedRunningTime="2026-07-27 21:03:50.478413682 +0000 UTC m=+22.610029904" Jul 27 21:03:50.492268 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:50.492208 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-npnjm" podStartSLOduration=2.952640461 podStartE2EDuration="21.492185947s" totalImagesPullingTime="18.539545486s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:03:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:03:50.491733411 +0000 UTC m=+22.623349634" watchObservedRunningTime="2026-07-27 21:03:50.492185947 +0000 UTC m=+22.623802171" Jul 27 21:03:50.511676 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:50.511617 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-hq8kb" podStartSLOduration=2.976993966 podStartE2EDuration="21.511598414s" totalImagesPullingTime="18.534604448s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:03:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:03:50.511389392 +0000 UTC m=+22.643005712" watchObservedRunningTime="2026-07-27 21:03:50.511598414 +0000 UTC m=+22.643214634" Jul 27 21:03:50.534493 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:50.534442 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-tv2n5" podStartSLOduration=2.972716456 podStartE2EDuration="21.53442542s" totalImagesPullingTime="18.561708964s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:03:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:03:50.534166016 +0000 UTC m=+22.665782230" watchObservedRunningTime="2026-07-27 21:03:50.53442542 +0000 UTC m=+22.666041643" Jul 27 21:03:51.360737 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:51.360646 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lrnv5" Jul 27 21:03:51.360737 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:51.360694 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lvg8z" Jul 27 21:03:51.361007 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:51.360785 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lrnv5" podUID="8157bc07-d223-44de-aee6-ffa0c683a2cb" Jul 27 21:03:51.361007 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:51.360931 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lvg8z" podUID="0c32154a-b41d-45c4-afa6-240a8ea24835" Jul 27 21:03:51.478079 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:51.478043 2577 generic.go:356] "Generic (PLEG): container finished" podID="bbf44a86-6e26-4584-b851-86bcbf8d10f9" containerID="a398c6192ef7d4cbbd85206dec9f3ce0d2b9bd5f5f81519df7f63bf4cd09ff8d" exitCode=0 Jul 27 21:03:51.478899 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:51.478129 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5kfcg" event={"ID":"bbf44a86-6e26-4584-b851-86bcbf8d10f9","Type":"ContainerDied","Data":"a398c6192ef7d4cbbd85206dec9f3ce0d2b9bd5f5f81519df7f63bf4cd09ff8d"} Jul 27 21:03:51.481101 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:51.480997 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6mhn_1907dc14-1132-4eb9-a5ee-675b43df6e6d/ovn-acl-logging/0.log" Jul 27 21:03:51.481367 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:51.481343 2577 generic.go:356] "Generic (PLEG): container finished" podID="1907dc14-1132-4eb9-a5ee-675b43df6e6d" containerID="6b61cd0c1791418f0a0767e0519ff0f12433bd123ee69da8845ea1670c390e67" exitCode=1 Jul 27 21:03:51.481463 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:51.481443 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" event={"ID":"1907dc14-1132-4eb9-a5ee-675b43df6e6d","Type":"ContainerStarted","Data":"4469c880664109fca8758641f692b06986a7750b53f6dad48884566aefad7674"} Jul 27 21:03:51.481536 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:51.481520 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" event={"ID":"1907dc14-1132-4eb9-a5ee-675b43df6e6d","Type":"ContainerStarted","Data":"75628a59fe99fe019de44118bc0fb74670cfe4caafc6961455329577dcec8568"} Jul 27 21:03:51.481613 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:51.481564 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" event={"ID":"1907dc14-1132-4eb9-a5ee-675b43df6e6d","Type":"ContainerStarted","Data":"d52641b0d36f571a4c4aceb6001a4eb02bf8aa4ac6114d8af717b1abd644a1f4"} Jul 27 21:03:51.481670 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:51.481626 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" event={"ID":"1907dc14-1132-4eb9-a5ee-675b43df6e6d","Type":"ContainerStarted","Data":"09b5288559e561ccbef54c813bcb861584786ededae77a90baae3bc9b2f9ae02"} Jul 27 21:03:51.481670 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:51.481660 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" event={"ID":"1907dc14-1132-4eb9-a5ee-675b43df6e6d","Type":"ContainerDied","Data":"6b61cd0c1791418f0a0767e0519ff0f12433bd123ee69da8845ea1670c390e67"} Jul 27 21:03:51.481771 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:51.481703 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" event={"ID":"1907dc14-1132-4eb9-a5ee-675b43df6e6d","Type":"ContainerStarted","Data":"2abdc52e2ebe1de1fad8f32e8340284b8e884404803e19cd5d45b4e0a258c6d6"} Jul 27 21:03:51.498358 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:51.498306 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-201.ec2.internal" podStartSLOduration=21.498287478 podStartE2EDuration="21.498287478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:03:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:03:50.585367064 +0000 UTC m=+22.716983288" watchObservedRunningTime="2026-07-27 21:03:51.498287478 +0000 UTC m=+23.629903684" Jul 27 21:03:51.644258 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:51.644229 2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 27 21:03:52.286769 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:52.286646 2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-27T21:03:51.644251967Z","UUID":"c3f9b352-2a35-4016-b11e-b1b4a6543153","Handler":null,"Name":"","Endpoint":""} Jul 27 21:03:52.288587 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:52.288547 2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 27 21:03:52.288738 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:52.288612 2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 27 21:03:52.360652 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:52.360613 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2czx" Jul 27 21:03:52.360834 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:52.360760 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2czx" podUID="c77555ba-462e-49c5-b8b0-e83987e8a2e3" Jul 27 21:03:52.487109 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:52.487063 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jz7ws" event={"ID":"2f0a4058-4490-42ed-ae54-82e7ab83d15d","Type":"ContainerStarted","Data":"eedf112450a2367928b5165323cde030127fe85105120f097a6e3ec8dcd253db"} Jul 27 21:03:52.489099 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:52.489063 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-64nks" event={"ID":"f8ef5264-95a3-4193-ae7d-f0f10a7733ab","Type":"ContainerStarted","Data":"1771375cebdeab4d82b7b3868c705f967f1f4fe708c11df990204b3364e0e594"} Jul 27 21:03:52.505715 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:52.505650 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-64nks" podStartSLOduration=4.203543657 podStartE2EDuration="23.50562819s" totalImagesPullingTime="19.302084533s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:03:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:03:52.505102152 +0000 UTC m=+24.636718378" watchObservedRunningTime="2026-07-27 21:03:52.50562819 +0000 UTC m=+24.637244417" Jul 27 21:03:53.361077 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:53.360851 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lrnv5" Jul 27 21:03:53.361233 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:53.360882 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lvg8z" Jul 27 21:03:53.361233 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:53.361187 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lrnv5" podUID="8157bc07-d223-44de-aee6-ffa0c683a2cb" Jul 27 21:03:53.361333 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:53.361274 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lvg8z" podUID="0c32154a-b41d-45c4-afa6-240a8ea24835" Jul 27 21:03:53.495030 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:53.494998 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6mhn_1907dc14-1132-4eb9-a5ee-675b43df6e6d/ovn-acl-logging/0.log" Jul 27 21:03:53.495491 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:53.495444 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" event={"ID":"1907dc14-1132-4eb9-a5ee-675b43df6e6d","Type":"ContainerStarted","Data":"1d0589140699ac2eb58d65c9a1ba7e8162ea7e02d7a9550f1ecf2dd24d12ec49"} Jul 27 21:03:53.498140 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:53.498105 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jz7ws" event={"ID":"2f0a4058-4490-42ed-ae54-82e7ab83d15d","Type":"ContainerStarted","Data":"d2080206ccf4fcb93d0b163246c577b384da3decc1de736e7dab49e9edf13b97"} Jul 27 21:03:53.517758 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:53.517699 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jz7ws" podStartSLOduration=3.685862736 podStartE2EDuration="24.517674161s" totalImagesPullingTime="20.831811425s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:03:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:03:53.5174241 +0000 UTC m=+25.649040323" watchObservedRunningTime="2026-07-27 21:03:53.517674161 +0000 UTC m=+25.649290385" Jul 27 21:03:54.360949 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:54.360914 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2czx" Jul 27 21:03:54.361165 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:54.361029 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2czx" podUID="c77555ba-462e-49c5-b8b0-e83987e8a2e3" Jul 27 21:03:55.360563 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:55.360523 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lrnv5" Jul 27 21:03:55.361158 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:55.360523 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lvg8z" Jul 27 21:03:55.361158 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:55.360667 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lrnv5" podUID="8157bc07-d223-44de-aee6-ffa0c683a2cb" Jul 27 21:03:55.361158 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:55.360783 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lvg8z" podUID="0c32154a-b41d-45c4-afa6-240a8ea24835" Jul 27 21:03:56.360705 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:56.360668 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2czx" Jul 27 21:03:56.361094 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:56.360816 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2czx" podUID="c77555ba-462e-49c5-b8b0-e83987e8a2e3" Jul 27 21:03:57.361006 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:57.360827 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lrnv5" Jul 27 21:03:57.361677 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:57.360882 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lvg8z" Jul 27 21:03:57.361677 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:57.361108 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lrnv5" podUID="8157bc07-d223-44de-aee6-ffa0c683a2cb" Jul 27 21:03:57.361677 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:57.361158 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lvg8z" podUID="0c32154a-b41d-45c4-afa6-240a8ea24835" Jul 27 21:03:57.508231 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:57.508200 2577 generic.go:356] "Generic (PLEG): container finished" podID="bbf44a86-6e26-4584-b851-86bcbf8d10f9" containerID="91f12cea8006c40e19619f929311ec3ccc1361b9bf8e8f017504988f98776b82" exitCode=0 Jul 27 21:03:57.508410 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:57.508289 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5kfcg" event={"ID":"bbf44a86-6e26-4584-b851-86bcbf8d10f9","Type":"ContainerDied","Data":"91f12cea8006c40e19619f929311ec3ccc1361b9bf8e8f017504988f98776b82"} Jul 27 21:03:57.512024 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:57.512004 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6mhn_1907dc14-1132-4eb9-a5ee-675b43df6e6d/ovn-acl-logging/0.log" Jul 27 21:03:57.512398 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:57.512367 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" event={"ID":"1907dc14-1132-4eb9-a5ee-675b43df6e6d","Type":"ContainerStarted","Data":"7d2620f85c018bba86a9aa7e29db72d22d8b51e5b6bca7672466b00b6828c47c"} Jul 27 21:03:57.512713 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:57.512682 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" Jul 27 21:03:57.512813 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:57.512763 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" Jul 27 21:03:57.512813 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:57.512811 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" Jul 27 21:03:57.512912 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:57.512891 2577 scope.go:117] "RemoveContainer" containerID="6b61cd0c1791418f0a0767e0519ff0f12433bd123ee69da8845ea1670c390e67" Jul 27 21:03:57.529346 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:57.529322 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" Jul 27 21:03:57.533188 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:57.533163 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" Jul 27 21:03:58.362086 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:58.362046 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2czx" Jul 27 21:03:58.362594 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:58.362155 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2czx" podUID="c77555ba-462e-49c5-b8b0-e83987e8a2e3" Jul 27 21:03:58.518591 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:58.518548 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6mhn_1907dc14-1132-4eb9-a5ee-675b43df6e6d/ovn-acl-logging/0.log" Jul 27 21:03:58.519038 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:58.519011 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" event={"ID":"1907dc14-1132-4eb9-a5ee-675b43df6e6d","Type":"ContainerStarted","Data":"d01ec226f48d9aab32fd5f7f8542f118e834353487456237104e4fcc3855e95d"} Jul 27 21:03:58.547934 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:58.547878 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" podStartSLOduration=10.656257208 podStartE2EDuration="29.547861887s" totalImagesPullingTime="18.891604679s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:03:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:03:58.547492899 +0000 UTC m=+30.679109133" watchObservedRunningTime="2026-07-27 21:03:58.547861887 +0000 UTC m=+30.679478109" Jul 27 21:03:58.952095 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:58.952017 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lrnv5"] Jul 27 21:03:58.952227 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:58.952160 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lrnv5" Jul 27 21:03:58.952288 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:58.952264 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lrnv5" podUID="8157bc07-d223-44de-aee6-ffa0c683a2cb" Jul 27 21:03:58.956836 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:58.956799 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-l2czx"] Jul 27 21:03:58.956991 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:58.956910 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2czx" Jul 27 21:03:58.957039 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:58.957018 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2czx" podUID="c77555ba-462e-49c5-b8b0-e83987e8a2e3" Jul 27 21:03:58.957500 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:58.957476 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lvg8z"] Jul 27 21:03:58.957618 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:58.957595 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lvg8z" Jul 27 21:03:58.957711 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:03:58.957684 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lvg8z" podUID="0c32154a-b41d-45c4-afa6-240a8ea24835" Jul 27 21:03:59.523476 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:59.523440 2577 generic.go:356] "Generic (PLEG): container finished" podID="bbf44a86-6e26-4584-b851-86bcbf8d10f9" containerID="4568356b9e544ef591bc9c0b5eb0f4cbce7dce2b80c429ecb596ca1b3573edd6" exitCode=0 Jul 27 21:03:59.523941 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:03:59.523560 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5kfcg" event={"ID":"bbf44a86-6e26-4584-b851-86bcbf8d10f9","Type":"ContainerDied","Data":"4568356b9e544ef591bc9c0b5eb0f4cbce7dce2b80c429ecb596ca1b3573edd6"} Jul 27 21:04:00.360962 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:00.360770 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lrnv5" Jul 27 21:04:00.361126 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:00.360772 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2czx" Jul 27 21:04:00.361126 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:00.361039 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lrnv5" podUID="8157bc07-d223-44de-aee6-ffa0c683a2cb" Jul 27 21:04:00.361126 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:00.360773 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lvg8z" Jul 27 21:04:00.361238 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:00.361126 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2czx" podUID="c77555ba-462e-49c5-b8b0-e83987e8a2e3" Jul 27 21:04:00.361238 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:00.361208 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lvg8z" podUID="0c32154a-b41d-45c4-afa6-240a8ea24835" Jul 27 21:04:00.527965 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:00.527933 2577 generic.go:356] "Generic (PLEG): container finished" podID="bbf44a86-6e26-4584-b851-86bcbf8d10f9" containerID="c8411cb61532071ceb225c3a1b0caa964e9ee398a758238a758cc0f7aa8932e8" exitCode=0 Jul 27 21:04:00.528421 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:00.527996 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5kfcg" event={"ID":"bbf44a86-6e26-4584-b851-86bcbf8d10f9","Type":"ContainerDied","Data":"c8411cb61532071ceb225c3a1b0caa964e9ee398a758238a758cc0f7aa8932e8"} Jul 27 21:04:02.002027 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:02.001993 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c77555ba-462e-49c5-b8b0-e83987e8a2e3-original-pull-secret\") pod \"global-pull-secret-syncer-l2czx\" (UID: \"c77555ba-462e-49c5-b8b0-e83987e8a2e3\") " pod="kube-system/global-pull-secret-syncer-l2czx" Jul 27 21:04:02.002536 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:02.002044 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c32154a-b41d-45c4-afa6-240a8ea24835-metrics-certs\") pod \"network-metrics-daemon-lvg8z\" (UID: \"0c32154a-b41d-45c4-afa6-240a8ea24835\") " pod="openshift-multus/network-metrics-daemon-lvg8z" Jul 27 21:04:02.002536 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:02.002178 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 21:04:02.002536 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:02.002257 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c77555ba-462e-49c5-b8b0-e83987e8a2e3-original-pull-secret podName:c77555ba-462e-49c5-b8b0-e83987e8a2e3 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:34.002241857 +0000 UTC m=+66.133858062 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c77555ba-462e-49c5-b8b0-e83987e8a2e3-original-pull-secret") pod "global-pull-secret-syncer-l2czx" (UID: "c77555ba-462e-49c5-b8b0-e83987e8a2e3") : object "kube-system"/"original-pull-secret" not registered Jul 27 21:04:02.002536 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:02.002183 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 21:04:02.002536 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:02.002321 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c32154a-b41d-45c4-afa6-240a8ea24835-metrics-certs podName:0c32154a-b41d-45c4-afa6-240a8ea24835 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:34.0023079 +0000 UTC m=+66.133924104 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0c32154a-b41d-45c4-afa6-240a8ea24835-metrics-certs") pod "network-metrics-daemon-lvg8z" (UID: "0c32154a-b41d-45c4-afa6-240a8ea24835") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 21:04:02.102774 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:02.102724 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft79h\" (UniqueName: \"kubernetes.io/projected/8157bc07-d223-44de-aee6-ffa0c683a2cb-kube-api-access-ft79h\") pod \"network-check-target-lrnv5\" (UID: \"8157bc07-d223-44de-aee6-ffa0c683a2cb\") " pod="openshift-network-diagnostics/network-check-target-lrnv5" Jul 27 21:04:02.102948 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:02.102913 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 21:04:02.102948 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:02.102940 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 21:04:02.103045 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:02.102954 2577 projected.go:196] Error preparing data for projected volume kube-api-access-ft79h for pod openshift-network-diagnostics/network-check-target-lrnv5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 21:04:02.103045 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:02.103026 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8157bc07-d223-44de-aee6-ffa0c683a2cb-kube-api-access-ft79h podName:8157bc07-d223-44de-aee6-ffa0c683a2cb nodeName:}" failed. No retries permitted until 2026-07-27 21:04:34.103006183 +0000 UTC m=+66.234622404 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-ft79h" (UniqueName: "kubernetes.io/projected/8157bc07-d223-44de-aee6-ffa0c683a2cb-kube-api-access-ft79h") pod "network-check-target-lrnv5" (UID: "8157bc07-d223-44de-aee6-ffa0c683a2cb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 21:04:02.360787 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:02.360696 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2czx" Jul 27 21:04:02.360787 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:02.360702 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lvg8z" Jul 27 21:04:02.361121 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:02.360710 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lrnv5" Jul 27 21:04:02.361121 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:02.360834 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l2czx" podUID="c77555ba-462e-49c5-b8b0-e83987e8a2e3" Jul 27 21:04:02.361121 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:02.360903 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lrnv5" podUID="8157bc07-d223-44de-aee6-ffa0c683a2cb" Jul 27 21:04:02.361121 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:02.360991 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lvg8z" podUID="0c32154a-b41d-45c4-afa6-240a8ea24835" Jul 27 21:04:04.225642 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.225611 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-201.ec2.internal" event="NodeReady" Jul 27 21:04:04.226268 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.225760 2577 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 27 21:04:04.260030 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.259991 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-75fb88d74f-jhdnw"] Jul 27 21:04:04.262363 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.262342 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75fb88d74f-jhdnw" Jul 27 21:04:04.264690 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.264668 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 27 21:04:04.264796 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.264675 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 27 21:04:04.265063 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.265042 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 27 21:04:04.265180 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.265162 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-l946f\"" Jul 27 21:04:04.272201 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.272177 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 27 21:04:04.273812 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.273788 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-ppf68"] Jul 27 21:04:04.275505 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.275487 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ppf68" Jul 27 21:04:04.279190 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.278719 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jkr4s\"" Jul 27 21:04:04.279190 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.278752 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 27 21:04:04.280879 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.280759 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 27 21:04:04.280879 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.280790 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-75fb88d74f-jhdnw"] Jul 27 21:04:04.282823 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.281713 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-7gnng"] Jul 27 21:04:04.283878 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.283857 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7gnng" Jul 27 21:04:04.288883 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.288861 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 27 21:04:04.289153 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.289135 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-xwcxr\"" Jul 27 21:04:04.289259 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.289203 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 27 21:04:04.289318 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.289143 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 27 21:04:04.291649 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.291393 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ppf68"] Jul 27 21:04:04.299081 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.299052 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7gnng"] Jul 27 21:04:04.360386 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.360346 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lrnv5" Jul 27 21:04:04.360582 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.360345 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2czx" Jul 27 21:04:04.360582 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.360345 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lvg8z" Jul 27 21:04:04.362937 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.362912 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 27 21:04:04.363073 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.362967 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 27 21:04:04.363073 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.363043 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ww7jp\"" Jul 27 21:04:04.363180 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.362968 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 27 21:04:04.363334 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.363314 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bztw5\"" Jul 27 21:04:04.363417 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.363355 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 27 21:04:04.423253 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.422977 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/eb9ca170-7ea1-45de-aca4-54b4731ed82d-registry-certificates\") pod \"image-registry-75fb88d74f-jhdnw\" (UID: \"eb9ca170-7ea1-45de-aca4-54b4731ed82d\") " pod="openshift-image-registry/image-registry-75fb88d74f-jhdnw" Jul 27 21:04:04.423253 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.423056 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cfnf\" (UniqueName: \"kubernetes.io/projected/597ea753-2547-4a21-9678-89da32128dcb-kube-api-access-8cfnf\") pod \"ingress-canary-7gnng\" (UID: \"597ea753-2547-4a21-9678-89da32128dcb\") " pod="openshift-ingress-canary/ingress-canary-7gnng" Jul 27 21:04:04.423253 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.423094 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eb9ca170-7ea1-45de-aca4-54b4731ed82d-trusted-ca\") pod \"image-registry-75fb88d74f-jhdnw\" (UID: \"eb9ca170-7ea1-45de-aca4-54b4731ed82d\") " pod="openshift-image-registry/image-registry-75fb88d74f-jhdnw" Jul 27 21:04:04.423253 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.423154 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/597ea753-2547-4a21-9678-89da32128dcb-cert\") pod \"ingress-canary-7gnng\" (UID: \"597ea753-2547-4a21-9678-89da32128dcb\") " pod="openshift-ingress-canary/ingress-canary-7gnng" Jul 27 21:04:04.423253 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.423192 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/932ad643-a059-44a4-90f8-41016fc1e5c0-metrics-tls\") pod \"dns-default-ppf68\" (UID: \"932ad643-a059-44a4-90f8-41016fc1e5c0\") " pod="openshift-dns/dns-default-ppf68" Jul 27 21:04:04.423253 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.423225 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/932ad643-a059-44a4-90f8-41016fc1e5c0-tmp-dir\") pod \"dns-default-ppf68\" (UID: \"932ad643-a059-44a4-90f8-41016fc1e5c0\") " pod="openshift-dns/dns-default-ppf68" Jul 27 21:04:04.423686 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.423267 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/eb9ca170-7ea1-45de-aca4-54b4731ed82d-image-registry-private-configuration\") pod \"image-registry-75fb88d74f-jhdnw\" (UID: \"eb9ca170-7ea1-45de-aca4-54b4731ed82d\") " pod="openshift-image-registry/image-registry-75fb88d74f-jhdnw" Jul 27 21:04:04.423686 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.423332 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eb9ca170-7ea1-45de-aca4-54b4731ed82d-registry-tls\") pod \"image-registry-75fb88d74f-jhdnw\" (UID: \"eb9ca170-7ea1-45de-aca4-54b4731ed82d\") " pod="openshift-image-registry/image-registry-75fb88d74f-jhdnw" Jul 27 21:04:04.423686 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.423366 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/eb9ca170-7ea1-45de-aca4-54b4731ed82d-ca-trust-extracted\") pod \"image-registry-75fb88d74f-jhdnw\" (UID: \"eb9ca170-7ea1-45de-aca4-54b4731ed82d\") " pod="openshift-image-registry/image-registry-75fb88d74f-jhdnw" Jul 27 21:04:04.423686 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.423394 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eb9ca170-7ea1-45de-aca4-54b4731ed82d-bound-sa-token\") pod \"image-registry-75fb88d74f-jhdnw\" (UID: \"eb9ca170-7ea1-45de-aca4-54b4731ed82d\") " pod="openshift-image-registry/image-registry-75fb88d74f-jhdnw" Jul 27 21:04:04.423686 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.423427 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s62d6\" (UniqueName: \"kubernetes.io/projected/eb9ca170-7ea1-45de-aca4-54b4731ed82d-kube-api-access-s62d6\") pod \"image-registry-75fb88d74f-jhdnw\" (UID: \"eb9ca170-7ea1-45de-aca4-54b4731ed82d\") " pod="openshift-image-registry/image-registry-75fb88d74f-jhdnw" Jul 27 21:04:04.423686 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.423500 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndnrm\" (UniqueName: \"kubernetes.io/projected/932ad643-a059-44a4-90f8-41016fc1e5c0-kube-api-access-ndnrm\") pod \"dns-default-ppf68\" (UID: \"932ad643-a059-44a4-90f8-41016fc1e5c0\") " pod="openshift-dns/dns-default-ppf68" Jul 27 21:04:04.423686 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.423538 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/eb9ca170-7ea1-45de-aca4-54b4731ed82d-installation-pull-secrets\") pod \"image-registry-75fb88d74f-jhdnw\" (UID: \"eb9ca170-7ea1-45de-aca4-54b4731ed82d\") " pod="openshift-image-registry/image-registry-75fb88d74f-jhdnw" Jul 27 21:04:04.423686 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.423592 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/932ad643-a059-44a4-90f8-41016fc1e5c0-config-volume\") pod \"dns-default-ppf68\" (UID: \"932ad643-a059-44a4-90f8-41016fc1e5c0\") " pod="openshift-dns/dns-default-ppf68" Jul 27 21:04:04.524510 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.524474 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/eb9ca170-7ea1-45de-aca4-54b4731ed82d-image-registry-private-configuration\") pod \"image-registry-75fb88d74f-jhdnw\" (UID: \"eb9ca170-7ea1-45de-aca4-54b4731ed82d\") " pod="openshift-image-registry/image-registry-75fb88d74f-jhdnw" Jul 27 21:04:04.524684 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.524525 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eb9ca170-7ea1-45de-aca4-54b4731ed82d-registry-tls\") pod \"image-registry-75fb88d74f-jhdnw\" (UID: \"eb9ca170-7ea1-45de-aca4-54b4731ed82d\") " pod="openshift-image-registry/image-registry-75fb88d74f-jhdnw" Jul 27 21:04:04.524684 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.524543 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/eb9ca170-7ea1-45de-aca4-54b4731ed82d-ca-trust-extracted\") pod \"image-registry-75fb88d74f-jhdnw\" (UID: \"eb9ca170-7ea1-45de-aca4-54b4731ed82d\") " pod="openshift-image-registry/image-registry-75fb88d74f-jhdnw" Jul 27 21:04:04.524684 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.524560 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eb9ca170-7ea1-45de-aca4-54b4731ed82d-bound-sa-token\") pod \"image-registry-75fb88d74f-jhdnw\" (UID: \"eb9ca170-7ea1-45de-aca4-54b4731ed82d\") " pod="openshift-image-registry/image-registry-75fb88d74f-jhdnw" Jul 27 21:04:04.524833 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:04.524676 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 27 21:04:04.524833 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:04.524700 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75fb88d74f-jhdnw: secret "image-registry-tls" not found Jul 27 21:04:04.524833 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:04.524762 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb9ca170-7ea1-45de-aca4-54b4731ed82d-registry-tls podName:eb9ca170-7ea1-45de-aca4-54b4731ed82d nodeName:}" failed. No retries permitted until 2026-07-27 21:04:05.024739677 +0000 UTC m=+37.156355898 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/eb9ca170-7ea1-45de-aca4-54b4731ed82d-registry-tls") pod "image-registry-75fb88d74f-jhdnw" (UID: "eb9ca170-7ea1-45de-aca4-54b4731ed82d") : secret "image-registry-tls" not found Jul 27 21:04:04.524833 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.524680 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s62d6\" (UniqueName: \"kubernetes.io/projected/eb9ca170-7ea1-45de-aca4-54b4731ed82d-kube-api-access-s62d6\") pod \"image-registry-75fb88d74f-jhdnw\" (UID: \"eb9ca170-7ea1-45de-aca4-54b4731ed82d\") " pod="openshift-image-registry/image-registry-75fb88d74f-jhdnw" Jul 27 21:04:04.525037 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.524863 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndnrm\" (UniqueName: \"kubernetes.io/projected/932ad643-a059-44a4-90f8-41016fc1e5c0-kube-api-access-ndnrm\") pod \"dns-default-ppf68\" (UID: \"932ad643-a059-44a4-90f8-41016fc1e5c0\") " pod="openshift-dns/dns-default-ppf68" Jul 27 21:04:04.525037 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.524892 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/eb9ca170-7ea1-45de-aca4-54b4731ed82d-installation-pull-secrets\") pod \"image-registry-75fb88d74f-jhdnw\" (UID: \"eb9ca170-7ea1-45de-aca4-54b4731ed82d\") " pod="openshift-image-registry/image-registry-75fb88d74f-jhdnw" Jul 27 21:04:04.525037 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.524921 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/932ad643-a059-44a4-90f8-41016fc1e5c0-config-volume\") pod \"dns-default-ppf68\" (UID: \"932ad643-a059-44a4-90f8-41016fc1e5c0\") " pod="openshift-dns/dns-default-ppf68" Jul 27 21:04:04.525037 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.524953 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/eb9ca170-7ea1-45de-aca4-54b4731ed82d-registry-certificates\") pod \"image-registry-75fb88d74f-jhdnw\" (UID: \"eb9ca170-7ea1-45de-aca4-54b4731ed82d\") " pod="openshift-image-registry/image-registry-75fb88d74f-jhdnw" Jul 27 21:04:04.525037 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.524956 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/eb9ca170-7ea1-45de-aca4-54b4731ed82d-ca-trust-extracted\") pod \"image-registry-75fb88d74f-jhdnw\" (UID: \"eb9ca170-7ea1-45de-aca4-54b4731ed82d\") " pod="openshift-image-registry/image-registry-75fb88d74f-jhdnw" Jul 27 21:04:04.525037 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.525005 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cfnf\" (UniqueName: \"kubernetes.io/projected/597ea753-2547-4a21-9678-89da32128dcb-kube-api-access-8cfnf\") pod \"ingress-canary-7gnng\" (UID: \"597ea753-2547-4a21-9678-89da32128dcb\") " pod="openshift-ingress-canary/ingress-canary-7gnng" Jul 27 21:04:04.525037 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.525026 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eb9ca170-7ea1-45de-aca4-54b4731ed82d-trusted-ca\") pod \"image-registry-75fb88d74f-jhdnw\" (UID: \"eb9ca170-7ea1-45de-aca4-54b4731ed82d\") " pod="openshift-image-registry/image-registry-75fb88d74f-jhdnw" Jul 27 21:04:04.525379 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.525150 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/597ea753-2547-4a21-9678-89da32128dcb-cert\") pod \"ingress-canary-7gnng\" (UID: \"597ea753-2547-4a21-9678-89da32128dcb\") " pod="openshift-ingress-canary/ingress-canary-7gnng" Jul 27 21:04:04.525379 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.525181 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/932ad643-a059-44a4-90f8-41016fc1e5c0-metrics-tls\") pod \"dns-default-ppf68\" (UID: \"932ad643-a059-44a4-90f8-41016fc1e5c0\") " pod="openshift-dns/dns-default-ppf68" Jul 27 21:04:04.525379 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.525234 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/932ad643-a059-44a4-90f8-41016fc1e5c0-tmp-dir\") pod \"dns-default-ppf68\" (UID: \"932ad643-a059-44a4-90f8-41016fc1e5c0\") " pod="openshift-dns/dns-default-ppf68" Jul 27 21:04:04.525379 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:04.525264 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 21:04:04.525379 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:04.525319 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/932ad643-a059-44a4-90f8-41016fc1e5c0-metrics-tls podName:932ad643-a059-44a4-90f8-41016fc1e5c0 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:05.02530318 +0000 UTC m=+37.156919402 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/932ad643-a059-44a4-90f8-41016fc1e5c0-metrics-tls") pod "dns-default-ppf68" (UID: "932ad643-a059-44a4-90f8-41016fc1e5c0") : secret "dns-default-metrics-tls" not found Jul 27 21:04:04.525655 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.525535 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/eb9ca170-7ea1-45de-aca4-54b4731ed82d-registry-certificates\") pod \"image-registry-75fb88d74f-jhdnw\" (UID: \"eb9ca170-7ea1-45de-aca4-54b4731ed82d\") " pod="openshift-image-registry/image-registry-75fb88d74f-jhdnw" Jul 27 21:04:04.525844 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.525819 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/932ad643-a059-44a4-90f8-41016fc1e5c0-config-volume\") pod \"dns-default-ppf68\" (UID: \"932ad643-a059-44a4-90f8-41016fc1e5c0\") " pod="openshift-dns/dns-default-ppf68" Jul 27 21:04:04.526047 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.526016 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/932ad643-a059-44a4-90f8-41016fc1e5c0-tmp-dir\") pod \"dns-default-ppf68\" (UID: \"932ad643-a059-44a4-90f8-41016fc1e5c0\") " pod="openshift-dns/dns-default-ppf68" Jul 27 21:04:04.526107 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.526068 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eb9ca170-7ea1-45de-aca4-54b4731ed82d-trusted-ca\") pod \"image-registry-75fb88d74f-jhdnw\" (UID: \"eb9ca170-7ea1-45de-aca4-54b4731ed82d\") " pod="openshift-image-registry/image-registry-75fb88d74f-jhdnw" Jul 27 21:04:04.529186 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.529157 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/eb9ca170-7ea1-45de-aca4-54b4731ed82d-image-registry-private-configuration\") pod \"image-registry-75fb88d74f-jhdnw\" (UID: \"eb9ca170-7ea1-45de-aca4-54b4731ed82d\") " pod="openshift-image-registry/image-registry-75fb88d74f-jhdnw" Jul 27 21:04:04.529287 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.529243 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/eb9ca170-7ea1-45de-aca4-54b4731ed82d-installation-pull-secrets\") pod \"image-registry-75fb88d74f-jhdnw\" (UID: \"eb9ca170-7ea1-45de-aca4-54b4731ed82d\") " pod="openshift-image-registry/image-registry-75fb88d74f-jhdnw" Jul 27 21:04:04.529287 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.529278 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/597ea753-2547-4a21-9678-89da32128dcb-cert\") pod \"ingress-canary-7gnng\" (UID: \"597ea753-2547-4a21-9678-89da32128dcb\") " pod="openshift-ingress-canary/ingress-canary-7gnng" Jul 27 21:04:04.533865 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.533834 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s62d6\" (UniqueName: \"kubernetes.io/projected/eb9ca170-7ea1-45de-aca4-54b4731ed82d-kube-api-access-s62d6\") pod \"image-registry-75fb88d74f-jhdnw\" (UID: \"eb9ca170-7ea1-45de-aca4-54b4731ed82d\") " pod="openshift-image-registry/image-registry-75fb88d74f-jhdnw" Jul 27 21:04:04.534132 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.534046 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eb9ca170-7ea1-45de-aca4-54b4731ed82d-bound-sa-token\") pod \"image-registry-75fb88d74f-jhdnw\" (UID: \"eb9ca170-7ea1-45de-aca4-54b4731ed82d\") " pod="openshift-image-registry/image-registry-75fb88d74f-jhdnw" Jul 27 21:04:04.534132 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.534086 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndnrm\" (UniqueName: \"kubernetes.io/projected/932ad643-a059-44a4-90f8-41016fc1e5c0-kube-api-access-ndnrm\") pod \"dns-default-ppf68\" (UID: \"932ad643-a059-44a4-90f8-41016fc1e5c0\") " pod="openshift-dns/dns-default-ppf68" Jul 27 21:04:04.534355 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.534309 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cfnf\" (UniqueName: \"kubernetes.io/projected/597ea753-2547-4a21-9678-89da32128dcb-kube-api-access-8cfnf\") pod \"ingress-canary-7gnng\" (UID: \"597ea753-2547-4a21-9678-89da32128dcb\") " pod="openshift-ingress-canary/ingress-canary-7gnng" Jul 27 21:04:04.594533 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.594488 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7gnng" Jul 27 21:04:04.760396 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:04.760144 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7gnng"] Jul 27 21:04:04.764524 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:04:04.764479 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod597ea753_2547_4a21_9678_89da32128dcb.slice/crio-b26307da363f3555e71fc114d0f54c2aae55402765522194d144ee55a082817f WatchSource:0}: Error finding container b26307da363f3555e71fc114d0f54c2aae55402765522194d144ee55a082817f: Status 404 returned error can't find the container with id b26307da363f3555e71fc114d0f54c2aae55402765522194d144ee55a082817f Jul 27 21:04:05.029520 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:05.029424 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eb9ca170-7ea1-45de-aca4-54b4731ed82d-registry-tls\") pod \"image-registry-75fb88d74f-jhdnw\" (UID: \"eb9ca170-7ea1-45de-aca4-54b4731ed82d\") " pod="openshift-image-registry/image-registry-75fb88d74f-jhdnw" Jul 27 21:04:05.029520 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:05.029489 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/932ad643-a059-44a4-90f8-41016fc1e5c0-metrics-tls\") pod \"dns-default-ppf68\" (UID: \"932ad643-a059-44a4-90f8-41016fc1e5c0\") " pod="openshift-dns/dns-default-ppf68" Jul 27 21:04:05.029773 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:05.029604 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 21:04:05.029773 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:05.029618 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 27 21:04:05.029773 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:05.029641 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75fb88d74f-jhdnw: secret "image-registry-tls" not found Jul 27 21:04:05.029773 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:05.029655 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/932ad643-a059-44a4-90f8-41016fc1e5c0-metrics-tls podName:932ad643-a059-44a4-90f8-41016fc1e5c0 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:06.029642588 +0000 UTC m=+38.161258788 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/932ad643-a059-44a4-90f8-41016fc1e5c0-metrics-tls") pod "dns-default-ppf68" (UID: "932ad643-a059-44a4-90f8-41016fc1e5c0") : secret "dns-default-metrics-tls" not found Jul 27 21:04:05.029773 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:05.029691 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb9ca170-7ea1-45de-aca4-54b4731ed82d-registry-tls podName:eb9ca170-7ea1-45de-aca4-54b4731ed82d nodeName:}" failed. No retries permitted until 2026-07-27 21:04:06.029675013 +0000 UTC m=+38.161291215 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/eb9ca170-7ea1-45de-aca4-54b4731ed82d-registry-tls") pod "image-registry-75fb88d74f-jhdnw" (UID: "eb9ca170-7ea1-45de-aca4-54b4731ed82d") : secret "image-registry-tls" not found Jul 27 21:04:05.543954 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:05.543915 2577 generic.go:356] "Generic (PLEG): container finished" podID="bbf44a86-6e26-4584-b851-86bcbf8d10f9" containerID="8a216b372e2ba90fdb94ffab3579ef6cbf3e1a9596cddb76544d0bc90a8ee08e" exitCode=0 Jul 27 21:04:05.544524 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:05.544001 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5kfcg" event={"ID":"bbf44a86-6e26-4584-b851-86bcbf8d10f9","Type":"ContainerDied","Data":"8a216b372e2ba90fdb94ffab3579ef6cbf3e1a9596cddb76544d0bc90a8ee08e"} Jul 27 21:04:05.545507 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:05.545476 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7gnng" event={"ID":"597ea753-2547-4a21-9678-89da32128dcb","Type":"ContainerStarted","Data":"b26307da363f3555e71fc114d0f54c2aae55402765522194d144ee55a082817f"} Jul 27 21:04:06.037936 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:06.037894 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/932ad643-a059-44a4-90f8-41016fc1e5c0-metrics-tls\") pod \"dns-default-ppf68\" (UID: \"932ad643-a059-44a4-90f8-41016fc1e5c0\") " pod="openshift-dns/dns-default-ppf68" Jul 27 21:04:06.038121 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:06.038049 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 21:04:06.038121 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:06.038072 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eb9ca170-7ea1-45de-aca4-54b4731ed82d-registry-tls\") pod \"image-registry-75fb88d74f-jhdnw\" (UID: \"eb9ca170-7ea1-45de-aca4-54b4731ed82d\") " pod="openshift-image-registry/image-registry-75fb88d74f-jhdnw" Jul 27 21:04:06.038121 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:06.038121 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/932ad643-a059-44a4-90f8-41016fc1e5c0-metrics-tls podName:932ad643-a059-44a4-90f8-41016fc1e5c0 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:08.038100089 +0000 UTC m=+40.169716302 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/932ad643-a059-44a4-90f8-41016fc1e5c0-metrics-tls") pod "dns-default-ppf68" (UID: "932ad643-a059-44a4-90f8-41016fc1e5c0") : secret "dns-default-metrics-tls" not found Jul 27 21:04:06.038283 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:06.038162 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 27 21:04:06.038283 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:06.038174 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75fb88d74f-jhdnw: secret "image-registry-tls" not found Jul 27 21:04:06.038283 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:06.038209 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb9ca170-7ea1-45de-aca4-54b4731ed82d-registry-tls podName:eb9ca170-7ea1-45de-aca4-54b4731ed82d nodeName:}" failed. No retries permitted until 2026-07-27 21:04:08.038197928 +0000 UTC m=+40.169814131 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/eb9ca170-7ea1-45de-aca4-54b4731ed82d-registry-tls") pod "image-registry-75fb88d74f-jhdnw" (UID: "eb9ca170-7ea1-45de-aca4-54b4731ed82d") : secret "image-registry-tls" not found Jul 27 21:04:06.551644 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:06.551606 2577 generic.go:356] "Generic (PLEG): container finished" podID="bbf44a86-6e26-4584-b851-86bcbf8d10f9" containerID="508e03c40dea23b3babb60b528deb1e089a73a460241dcdfa0eb1eb42b163e39" exitCode=0 Jul 27 21:04:06.552046 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:06.551651 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5kfcg" event={"ID":"bbf44a86-6e26-4584-b851-86bcbf8d10f9","Type":"ContainerDied","Data":"508e03c40dea23b3babb60b528deb1e089a73a460241dcdfa0eb1eb42b163e39"} Jul 27 21:04:07.557518 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:07.557479 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5kfcg" event={"ID":"bbf44a86-6e26-4584-b851-86bcbf8d10f9","Type":"ContainerStarted","Data":"b3e25b35f73bb3550b8ba101818ae440bf68b30624911c8e3263121b855a2869"} Jul 27 21:04:07.583932 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:07.583871 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-5kfcg" podStartSLOduration=8.49392654 podStartE2EDuration="38.58385013s" totalImagesPullingTime="30.08992359s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:03:29 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:07.58226908 +0000 UTC m=+39.713885390" watchObservedRunningTime="2026-07-27 21:04:07.58385013 +0000 UTC m=+39.715466355" Jul 27 21:04:08.054293 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:08.054247 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/932ad643-a059-44a4-90f8-41016fc1e5c0-metrics-tls\") pod \"dns-default-ppf68\" (UID: \"932ad643-a059-44a4-90f8-41016fc1e5c0\") " pod="openshift-dns/dns-default-ppf68" Jul 27 21:04:08.054466 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:08.054315 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eb9ca170-7ea1-45de-aca4-54b4731ed82d-registry-tls\") pod \"image-registry-75fb88d74f-jhdnw\" (UID: \"eb9ca170-7ea1-45de-aca4-54b4731ed82d\") " pod="openshift-image-registry/image-registry-75fb88d74f-jhdnw" Jul 27 21:04:08.054466 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:08.054401 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 21:04:08.054466 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:08.054404 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 27 21:04:08.054466 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:08.054437 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75fb88d74f-jhdnw: secret "image-registry-tls" not found Jul 27 21:04:08.054657 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:08.054477 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/932ad643-a059-44a4-90f8-41016fc1e5c0-metrics-tls podName:932ad643-a059-44a4-90f8-41016fc1e5c0 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:12.054461959 +0000 UTC m=+44.186078159 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/932ad643-a059-44a4-90f8-41016fc1e5c0-metrics-tls") pod "dns-default-ppf68" (UID: "932ad643-a059-44a4-90f8-41016fc1e5c0") : secret "dns-default-metrics-tls" not found Jul 27 21:04:08.054657 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:08.054494 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb9ca170-7ea1-45de-aca4-54b4731ed82d-registry-tls podName:eb9ca170-7ea1-45de-aca4-54b4731ed82d nodeName:}" failed. No retries permitted until 2026-07-27 21:04:12.05448775 +0000 UTC m=+44.186103952 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/eb9ca170-7ea1-45de-aca4-54b4731ed82d-registry-tls") pod "image-registry-75fb88d74f-jhdnw" (UID: "eb9ca170-7ea1-45de-aca4-54b4731ed82d") : secret "image-registry-tls" not found Jul 27 21:04:08.561643 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:08.561599 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7gnng" event={"ID":"597ea753-2547-4a21-9678-89da32128dcb","Type":"ContainerStarted","Data":"73037653caca0f08916607f8d88fc6c43cae56890c913a0dfb73665cd66b5fb5"} Jul 27 21:04:08.579515 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:08.579460 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-7gnng" podStartSLOduration=1.8261873610000001 podStartE2EDuration="4.579445288s" totalImagesPullingTime="2.753257927s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:04:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:08.577830633 +0000 UTC m=+40.709446856" watchObservedRunningTime="2026-07-27 21:04:08.579445288 +0000 UTC m=+40.711061580" Jul 27 21:04:12.086258 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:12.086219 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eb9ca170-7ea1-45de-aca4-54b4731ed82d-registry-tls\") pod \"image-registry-75fb88d74f-jhdnw\" (UID: \"eb9ca170-7ea1-45de-aca4-54b4731ed82d\") " pod="openshift-image-registry/image-registry-75fb88d74f-jhdnw" Jul 27 21:04:12.086720 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:12.086361 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/932ad643-a059-44a4-90f8-41016fc1e5c0-metrics-tls\") pod \"dns-default-ppf68\" (UID: \"932ad643-a059-44a4-90f8-41016fc1e5c0\") " pod="openshift-dns/dns-default-ppf68" Jul 27 21:04:12.086720 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:12.086378 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 27 21:04:12.086720 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:12.086400 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75fb88d74f-jhdnw: secret "image-registry-tls" not found Jul 27 21:04:12.086720 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:12.086448 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 21:04:12.086720 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:12.086458 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb9ca170-7ea1-45de-aca4-54b4731ed82d-registry-tls podName:eb9ca170-7ea1-45de-aca4-54b4731ed82d nodeName:}" failed. No retries permitted until 2026-07-27 21:04:20.086441594 +0000 UTC m=+52.218057799 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/eb9ca170-7ea1-45de-aca4-54b4731ed82d-registry-tls") pod "image-registry-75fb88d74f-jhdnw" (UID: "eb9ca170-7ea1-45de-aca4-54b4731ed82d") : secret "image-registry-tls" not found Jul 27 21:04:12.086720 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:12.086489 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/932ad643-a059-44a4-90f8-41016fc1e5c0-metrics-tls podName:932ad643-a059-44a4-90f8-41016fc1e5c0 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:20.086478181 +0000 UTC m=+52.218094382 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/932ad643-a059-44a4-90f8-41016fc1e5c0-metrics-tls") pod "dns-default-ppf68" (UID: "932ad643-a059-44a4-90f8-41016fc1e5c0") : secret "dns-default-metrics-tls" not found Jul 27 21:04:13.105421 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.105384 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-gdgs7"] Jul 27 21:04:13.138991 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.138955 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-trwhz"] Jul 27 21:04:13.139126 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.139078 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-gdgs7" Jul 27 21:04:13.142129 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.142100 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-mlrfk\"" Jul 27 21:04:13.156828 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.156798 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-64c559777f-75n27"] Jul 27 21:04:13.156995 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.156928 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-trwhz" Jul 27 21:04:13.162041 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.162013 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 27 21:04:13.163202 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.163167 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 27 21:04:13.163307 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.163176 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 27 21:04:13.163307 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.163297 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-rr8x9\"" Jul 27 21:04:13.163418 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.163306 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 27 21:04:13.170841 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.170820 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-gdgs7"] Jul 27 21:04:13.170929 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.170849 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-trwhz"] Jul 27 21:04:13.170929 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.170862 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-64c559777f-75n27"] Jul 27 21:04:13.171003 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.170942 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-64c559777f-75n27" Jul 27 21:04:13.173744 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.173720 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 27 21:04:13.173744 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.173731 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 27 21:04:13.173744 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.173737 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 27 21:04:13.174079 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.174063 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 27 21:04:13.174115 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.174100 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 27 21:04:13.174427 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.174412 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-4zpl8\"" Jul 27 21:04:13.174543 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.174529 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 27 21:04:13.247668 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.247634 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-9rp9b"] Jul 27 21:04:13.264863 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.264826 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-dm69z"] Jul 27 21:04:13.265027 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.264962 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-9rp9b" Jul 27 21:04:13.269504 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.269480 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 27 21:04:13.269658 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.269481 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 27 21:04:13.269753 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.269734 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-jp48s\"" Jul 27 21:04:13.269977 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.269953 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 27 21:04:13.270082 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.270052 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 27 21:04:13.278852 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.278827 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-9qzp5"] Jul 27 21:04:13.278962 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.278948 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-dm69z" Jul 27 21:04:13.282515 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.282490 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 27 21:04:13.282681 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.282520 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 27 21:04:13.282681 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.282487 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-5r8h2\"" Jul 27 21:04:13.282681 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.282538 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 27 21:04:13.282681 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.282526 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 27 21:04:13.296918 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.296891 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-9qzp5"] Jul 27 21:04:13.296918 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.296923 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-dm69z"] Jul 27 21:04:13.297075 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.296938 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-9rp9b"] Jul 27 21:04:13.297075 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.297038 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-9qzp5" Jul 27 21:04:13.297144 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.297096 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86e4b5a1-4521-43b3-9bf8-c42ec92d21f8-service-ca-bundle\") pod \"router-default-64c559777f-75n27\" (UID: \"86e4b5a1-4521-43b3-9bf8-c42ec92d21f8\") " pod="openshift-ingress/router-default-64c559777f-75n27" Jul 27 21:04:13.297144 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.297126 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/79a93222-a9cb-4baf-8113-8e46390ff478-serving-cert\") pod \"service-ca-operator-5d6c75f84f-trwhz\" (UID: \"79a93222-a9cb-4baf-8113-8e46390ff478\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-trwhz" Jul 27 21:04:13.297202 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.297178 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/86e4b5a1-4521-43b3-9bf8-c42ec92d21f8-stats-auth\") pod \"router-default-64c559777f-75n27\" (UID: \"86e4b5a1-4521-43b3-9bf8-c42ec92d21f8\") " pod="openshift-ingress/router-default-64c559777f-75n27" Jul 27 21:04:13.297241 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.297203 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2fkj\" (UniqueName: \"kubernetes.io/projected/86e4b5a1-4521-43b3-9bf8-c42ec92d21f8-kube-api-access-p2fkj\") pod \"router-default-64c559777f-75n27\" (UID: \"86e4b5a1-4521-43b3-9bf8-c42ec92d21f8\") " pod="openshift-ingress/router-default-64c559777f-75n27" Jul 27 21:04:13.297271 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.297241 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/79a93222-a9cb-4baf-8113-8e46390ff478-config\") pod \"service-ca-operator-5d6c75f84f-trwhz\" (UID: \"79a93222-a9cb-4baf-8113-8e46390ff478\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-trwhz" Jul 27 21:04:13.297303 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.297289 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/86e4b5a1-4521-43b3-9bf8-c42ec92d21f8-default-certificate\") pod \"router-default-64c559777f-75n27\" (UID: \"86e4b5a1-4521-43b3-9bf8-c42ec92d21f8\") " pod="openshift-ingress/router-default-64c559777f-75n27" Jul 27 21:04:13.297332 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.297317 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjdq5\" (UniqueName: \"kubernetes.io/projected/ce9b0c11-99b0-4de1-bd3b-b5661465bd95-kube-api-access-tjdq5\") pod \"network-check-source-7d88cb55d8-gdgs7\" (UID: \"ce9b0c11-99b0-4de1-bd3b-b5661465bd95\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-gdgs7" Jul 27 21:04:13.297392 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.297380 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86e4b5a1-4521-43b3-9bf8-c42ec92d21f8-metrics-certs\") pod \"router-default-64c559777f-75n27\" (UID: \"86e4b5a1-4521-43b3-9bf8-c42ec92d21f8\") " pod="openshift-ingress/router-default-64c559777f-75n27" Jul 27 21:04:13.297495 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.297477 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpf8c\" (UniqueName: \"kubernetes.io/projected/79a93222-a9cb-4baf-8113-8e46390ff478-kube-api-access-mpf8c\") pod \"service-ca-operator-5d6c75f84f-trwhz\" (UID: \"79a93222-a9cb-4baf-8113-8e46390ff478\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-trwhz" Jul 27 21:04:13.299222 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.299199 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-68lkz\"" Jul 27 21:04:13.300718 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.300698 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 27 21:04:13.303677 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.303661 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 27 21:04:13.373261 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.373168 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-kwqx9"] Jul 27 21:04:13.389078 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.389044 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-zwmht"] Jul 27 21:04:13.389298 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.389267 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-kwqx9" Jul 27 21:04:13.397773 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.397750 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86e4b5a1-4521-43b3-9bf8-c42ec92d21f8-service-ca-bundle\") pod \"router-default-64c559777f-75n27\" (UID: \"86e4b5a1-4521-43b3-9bf8-c42ec92d21f8\") " pod="openshift-ingress/router-default-64c559777f-75n27" Jul 27 21:04:13.397883 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.397780 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/79a93222-a9cb-4baf-8113-8e46390ff478-serving-cert\") pod \"service-ca-operator-5d6c75f84f-trwhz\" (UID: \"79a93222-a9cb-4baf-8113-8e46390ff478\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-trwhz" Jul 27 21:04:13.397883 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.397802 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmzwr\" (UniqueName: \"kubernetes.io/projected/91a29cde-048a-4e5e-a804-5854f86081ed-kube-api-access-cmzwr\") pod \"cluster-monitoring-operator-548b489cb8-dm69z\" (UID: \"91a29cde-048a-4e5e-a804-5854f86081ed\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-dm69z" Jul 27 21:04:13.397883 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.397855 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/91a29cde-048a-4e5e-a804-5854f86081ed-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-dm69z\" (UID: \"91a29cde-048a-4e5e-a804-5854f86081ed\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-dm69z" Jul 27 21:04:13.398032 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.397892 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2fkj\" (UniqueName: \"kubernetes.io/projected/86e4b5a1-4521-43b3-9bf8-c42ec92d21f8-kube-api-access-p2fkj\") pod \"router-default-64c559777f-75n27\" (UID: \"86e4b5a1-4521-43b3-9bf8-c42ec92d21f8\") " pod="openshift-ingress/router-default-64c559777f-75n27" Jul 27 21:04:13.398032 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:13.397928 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/86e4b5a1-4521-43b3-9bf8-c42ec92d21f8-service-ca-bundle podName:86e4b5a1-4521-43b3-9bf8-c42ec92d21f8 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:13.897908038 +0000 UTC m=+46.029524269 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/86e4b5a1-4521-43b3-9bf8-c42ec92d21f8-service-ca-bundle") pod "router-default-64c559777f-75n27" (UID: "86e4b5a1-4521-43b3-9bf8-c42ec92d21f8") : configmap references non-existent config key: service-ca.crt Jul 27 21:04:13.398032 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.397945 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b2e7d196-079c-4bcd-aaf0-affbae8a0594-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-9rp9b\" (UID: \"b2e7d196-079c-4bcd-aaf0-affbae8a0594\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-9rp9b" Jul 27 21:04:13.398032 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.397965 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/86e4b5a1-4521-43b3-9bf8-c42ec92d21f8-default-certificate\") pod \"router-default-64c559777f-75n27\" (UID: \"86e4b5a1-4521-43b3-9bf8-c42ec92d21f8\") " pod="openshift-ingress/router-default-64c559777f-75n27" Jul 27 21:04:13.398032 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.397991 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjdq5\" (UniqueName: \"kubernetes.io/projected/ce9b0c11-99b0-4de1-bd3b-b5661465bd95-kube-api-access-tjdq5\") pod \"network-check-source-7d88cb55d8-gdgs7\" (UID: \"ce9b0c11-99b0-4de1-bd3b-b5661465bd95\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-gdgs7" Jul 27 21:04:13.398032 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.398016 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qpck\" (UniqueName: \"kubernetes.io/projected/b2e7d196-079c-4bcd-aaf0-affbae8a0594-kube-api-access-8qpck\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-9rp9b\" (UID: \"b2e7d196-079c-4bcd-aaf0-affbae8a0594\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-9rp9b" Jul 27 21:04:13.398292 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.398061 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpf8c\" (UniqueName: \"kubernetes.io/projected/79a93222-a9cb-4baf-8113-8e46390ff478-kube-api-access-mpf8c\") pod \"service-ca-operator-5d6c75f84f-trwhz\" (UID: \"79a93222-a9cb-4baf-8113-8e46390ff478\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-trwhz" Jul 27 21:04:13.398292 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.398085 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b2e7d196-079c-4bcd-aaf0-affbae8a0594-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-9rp9b\" (UID: \"b2e7d196-079c-4bcd-aaf0-affbae8a0594\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-9rp9b" Jul 27 21:04:13.398292 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.398112 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/91a29cde-048a-4e5e-a804-5854f86081ed-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-dm69z\" (UID: \"91a29cde-048a-4e5e-a804-5854f86081ed\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-dm69z" Jul 27 21:04:13.398292 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.398160 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/86e4b5a1-4521-43b3-9bf8-c42ec92d21f8-stats-auth\") pod \"router-default-64c559777f-75n27\" (UID: \"86e4b5a1-4521-43b3-9bf8-c42ec92d21f8\") " pod="openshift-ingress/router-default-64c559777f-75n27" Jul 27 21:04:13.398292 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.398198 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/79a93222-a9cb-4baf-8113-8e46390ff478-config\") pod \"service-ca-operator-5d6c75f84f-trwhz\" (UID: \"79a93222-a9cb-4baf-8113-8e46390ff478\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-trwhz" Jul 27 21:04:13.398292 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.398249 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx6qv\" (UniqueName: \"kubernetes.io/projected/a4fc83fe-7a79-4461-9d98-c93af693d74f-kube-api-access-mx6qv\") pod \"volume-data-source-validator-549c6d54cf-9qzp5\" (UID: \"a4fc83fe-7a79-4461-9d98-c93af693d74f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-9qzp5" Jul 27 21:04:13.398292 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.398286 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86e4b5a1-4521-43b3-9bf8-c42ec92d21f8-metrics-certs\") pod \"router-default-64c559777f-75n27\" (UID: \"86e4b5a1-4521-43b3-9bf8-c42ec92d21f8\") " pod="openshift-ingress/router-default-64c559777f-75n27" Jul 27 21:04:13.398591 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:13.398443 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 27 21:04:13.398591 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:13.398507 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86e4b5a1-4521-43b3-9bf8-c42ec92d21f8-metrics-certs podName:86e4b5a1-4521-43b3-9bf8-c42ec92d21f8 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:13.898491633 +0000 UTC m=+46.030107837 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86e4b5a1-4521-43b3-9bf8-c42ec92d21f8-metrics-certs") pod "router-default-64c559777f-75n27" (UID: "86e4b5a1-4521-43b3-9bf8-c42ec92d21f8") : secret "router-metrics-certs-default" not found Jul 27 21:04:13.398891 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.398870 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/79a93222-a9cb-4baf-8113-8e46390ff478-config\") pod \"service-ca-operator-5d6c75f84f-trwhz\" (UID: \"79a93222-a9cb-4baf-8113-8e46390ff478\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-trwhz" Jul 27 21:04:13.399171 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.399059 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-d2494"] Jul 27 21:04:13.399283 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.399254 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-zwmht" Jul 27 21:04:13.401801 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.401776 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 27 21:04:13.401917 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.401785 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 27 21:04:13.402043 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.402024 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/79a93222-a9cb-4baf-8113-8e46390ff478-serving-cert\") pod \"service-ca-operator-5d6c75f84f-trwhz\" (UID: \"79a93222-a9cb-4baf-8113-8e46390ff478\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-trwhz" Jul 27 21:04:13.402161 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.402141 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 27 21:04:13.402263 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.402144 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/86e4b5a1-4521-43b3-9bf8-c42ec92d21f8-stats-auth\") pod \"router-default-64c559777f-75n27\" (UID: \"86e4b5a1-4521-43b3-9bf8-c42ec92d21f8\") " pod="openshift-ingress/router-default-64c559777f-75n27" Jul 27 21:04:13.402263 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.402169 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 27 21:04:13.402372 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.402297 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/86e4b5a1-4521-43b3-9bf8-c42ec92d21f8-default-certificate\") pod \"router-default-64c559777f-75n27\" (UID: \"86e4b5a1-4521-43b3-9bf8-c42ec92d21f8\") " pod="openshift-ingress/router-default-64c559777f-75n27" Jul 27 21:04:13.402563 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.402544 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-cr2mk\"" Jul 27 21:04:13.402672 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.402614 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 27 21:04:13.402821 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.402798 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-fxz59\"" Jul 27 21:04:13.402933 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.402914 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 27 21:04:13.402997 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.402914 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 27 21:04:13.410200 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.410175 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 27 21:04:13.413364 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.413270 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-j5rmn"] Jul 27 21:04:13.413444 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.413379 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-d2494" Jul 27 21:04:13.416521 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.416502 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-bj9tf\"" Jul 27 21:04:13.417085 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.417069 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 27 21:04:13.417829 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.417815 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 27 21:04:13.425445 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.425424 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-zwmht"] Jul 27 21:04:13.425533 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.425458 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-kwqx9"] Jul 27 21:04:13.425533 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.425472 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-d2494"] Jul 27 21:04:13.425533 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.425485 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-j5rmn"] Jul 27 21:04:13.425711 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.425556 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-j5rmn" Jul 27 21:04:13.428236 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.428215 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpf8c\" (UniqueName: \"kubernetes.io/projected/79a93222-a9cb-4baf-8113-8e46390ff478-kube-api-access-mpf8c\") pod \"service-ca-operator-5d6c75f84f-trwhz\" (UID: \"79a93222-a9cb-4baf-8113-8e46390ff478\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-trwhz" Jul 27 21:04:13.428483 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.428463 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 27 21:04:13.428637 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.428487 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 27 21:04:13.428637 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.428628 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-gz2tn\"" Jul 27 21:04:13.428786 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.428591 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2fkj\" (UniqueName: \"kubernetes.io/projected/86e4b5a1-4521-43b3-9bf8-c42ec92d21f8-kube-api-access-p2fkj\") pod \"router-default-64c559777f-75n27\" (UID: \"86e4b5a1-4521-43b3-9bf8-c42ec92d21f8\") " pod="openshift-ingress/router-default-64c559777f-75n27" Jul 27 21:04:13.428786 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.428550 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 27 21:04:13.428866 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.428809 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjdq5\" (UniqueName: \"kubernetes.io/projected/ce9b0c11-99b0-4de1-bd3b-b5661465bd95-kube-api-access-tjdq5\") pod \"network-check-source-7d88cb55d8-gdgs7\" (UID: \"ce9b0c11-99b0-4de1-bd3b-b5661465bd95\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-gdgs7" Jul 27 21:04:13.428987 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.428972 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 27 21:04:13.435742 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.435723 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 27 21:04:13.447651 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.447625 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-gdgs7" Jul 27 21:04:13.465227 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.465206 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-trwhz" Jul 27 21:04:13.500257 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.500137 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5df7w\" (UniqueName: \"kubernetes.io/projected/f33ee287-cfc5-436d-93cd-13801c2c522b-kube-api-access-5df7w\") pod \"insights-operator-5885fdb759-kwqx9\" (UID: \"f33ee287-cfc5-436d-93cd-13801c2c522b\") " pod="openshift-insights/insights-operator-5885fdb759-kwqx9" Jul 27 21:04:13.500257 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.500225 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d1562fc4-9850-4084-b2d9-7b388a637f32-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-zwmht\" (UID: \"d1562fc4-9850-4084-b2d9-7b388a637f32\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-zwmht" Jul 27 21:04:13.500257 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.500252 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f33ee287-cfc5-436d-93cd-13801c2c522b-snapshots\") pod \"insights-operator-5885fdb759-kwqx9\" (UID: \"f33ee287-cfc5-436d-93cd-13801c2c522b\") " pod="openshift-insights/insights-operator-5885fdb759-kwqx9" Jul 27 21:04:13.500527 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.500286 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx6qv\" (UniqueName: \"kubernetes.io/projected/a4fc83fe-7a79-4461-9d98-c93af693d74f-kube-api-access-mx6qv\") pod \"volume-data-source-validator-549c6d54cf-9qzp5\" (UID: \"a4fc83fe-7a79-4461-9d98-c93af693d74f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-9qzp5" Jul 27 21:04:13.500527 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.500363 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f33ee287-cfc5-436d-93cd-13801c2c522b-tmp\") pod \"insights-operator-5885fdb759-kwqx9\" (UID: \"f33ee287-cfc5-436d-93cd-13801c2c522b\") " pod="openshift-insights/insights-operator-5885fdb759-kwqx9" Jul 27 21:04:13.500527 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.500395 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f33ee287-cfc5-436d-93cd-13801c2c522b-service-ca-bundle\") pod \"insights-operator-5885fdb759-kwqx9\" (UID: \"f33ee287-cfc5-436d-93cd-13801c2c522b\") " pod="openshift-insights/insights-operator-5885fdb759-kwqx9" Jul 27 21:04:13.500527 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.500423 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmzwr\" (UniqueName: \"kubernetes.io/projected/91a29cde-048a-4e5e-a804-5854f86081ed-kube-api-access-cmzwr\") pod \"cluster-monitoring-operator-548b489cb8-dm69z\" (UID: \"91a29cde-048a-4e5e-a804-5854f86081ed\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-dm69z" Jul 27 21:04:13.500527 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.500453 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/91a29cde-048a-4e5e-a804-5854f86081ed-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-dm69z\" (UID: \"91a29cde-048a-4e5e-a804-5854f86081ed\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-dm69z" Jul 27 21:04:13.500527 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.500484 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b2e7d196-079c-4bcd-aaf0-affbae8a0594-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-9rp9b\" (UID: \"b2e7d196-079c-4bcd-aaf0-affbae8a0594\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-9rp9b" Jul 27 21:04:13.500527 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.500524 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qpck\" (UniqueName: \"kubernetes.io/projected/b2e7d196-079c-4bcd-aaf0-affbae8a0594-kube-api-access-8qpck\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-9rp9b\" (UID: \"b2e7d196-079c-4bcd-aaf0-affbae8a0594\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-9rp9b" Jul 27 21:04:13.500904 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.500557 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f33ee287-cfc5-436d-93cd-13801c2c522b-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-kwqx9\" (UID: \"f33ee287-cfc5-436d-93cd-13801c2c522b\") " pod="openshift-insights/insights-operator-5885fdb759-kwqx9" Jul 27 21:04:13.500904 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.500760 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b2e7d196-079c-4bcd-aaf0-affbae8a0594-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-9rp9b\" (UID: \"b2e7d196-079c-4bcd-aaf0-affbae8a0594\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-9rp9b" Jul 27 21:04:13.500904 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.500788 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f33ee287-cfc5-436d-93cd-13801c2c522b-serving-cert\") pod \"insights-operator-5885fdb759-kwqx9\" (UID: \"f33ee287-cfc5-436d-93cd-13801c2c522b\") " pod="openshift-insights/insights-operator-5885fdb759-kwqx9" Jul 27 21:04:13.500904 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.500811 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/91a29cde-048a-4e5e-a804-5854f86081ed-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-dm69z\" (UID: \"91a29cde-048a-4e5e-a804-5854f86081ed\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-dm69z" Jul 27 21:04:13.500904 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.500842 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlc9v\" (UniqueName: \"kubernetes.io/projected/d1562fc4-9850-4084-b2d9-7b388a637f32-kube-api-access-jlc9v\") pod \"cluster-samples-operator-6748489656-zwmht\" (UID: \"d1562fc4-9850-4084-b2d9-7b388a637f32\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-zwmht" Jul 27 21:04:13.501147 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:13.500926 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 27 21:04:13.501147 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:13.500992 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91a29cde-048a-4e5e-a804-5854f86081ed-cluster-monitoring-operator-tls podName:91a29cde-048a-4e5e-a804-5854f86081ed nodeName:}" failed. No retries permitted until 2026-07-27 21:04:14.000973647 +0000 UTC m=+46.132589849 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/91a29cde-048a-4e5e-a804-5854f86081ed-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-dm69z" (UID: "91a29cde-048a-4e5e-a804-5854f86081ed") : secret "cluster-monitoring-operator-tls" not found Jul 27 21:04:13.501314 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.501171 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b2e7d196-079c-4bcd-aaf0-affbae8a0594-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-9rp9b\" (UID: \"b2e7d196-079c-4bcd-aaf0-affbae8a0594\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-9rp9b" Jul 27 21:04:13.501314 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.501280 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/91a29cde-048a-4e5e-a804-5854f86081ed-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-dm69z\" (UID: \"91a29cde-048a-4e5e-a804-5854f86081ed\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-dm69z" Jul 27 21:04:13.503668 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.503644 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b2e7d196-079c-4bcd-aaf0-affbae8a0594-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-9rp9b\" (UID: \"b2e7d196-079c-4bcd-aaf0-affbae8a0594\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-9rp9b" Jul 27 21:04:13.511353 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.511321 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qpck\" (UniqueName: \"kubernetes.io/projected/b2e7d196-079c-4bcd-aaf0-affbae8a0594-kube-api-access-8qpck\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-9rp9b\" (UID: \"b2e7d196-079c-4bcd-aaf0-affbae8a0594\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-9rp9b" Jul 27 21:04:13.511555 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.511529 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx6qv\" (UniqueName: \"kubernetes.io/projected/a4fc83fe-7a79-4461-9d98-c93af693d74f-kube-api-access-mx6qv\") pod \"volume-data-source-validator-549c6d54cf-9qzp5\" (UID: \"a4fc83fe-7a79-4461-9d98-c93af693d74f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-9qzp5" Jul 27 21:04:13.512082 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.512015 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmzwr\" (UniqueName: \"kubernetes.io/projected/91a29cde-048a-4e5e-a804-5854f86081ed-kube-api-access-cmzwr\") pod \"cluster-monitoring-operator-548b489cb8-dm69z\" (UID: \"91a29cde-048a-4e5e-a804-5854f86081ed\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-dm69z" Jul 27 21:04:13.575805 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.575746 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-9rp9b" Jul 27 21:04:13.599774 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.599743 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-gdgs7"] Jul 27 21:04:13.601794 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.601768 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e066fd54-5796-438f-bdfc-784986a3713c-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-d2494\" (UID: \"e066fd54-5796-438f-bdfc-784986a3713c\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-d2494" Jul 27 21:04:13.601925 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.601821 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rdxt\" (UniqueName: \"kubernetes.io/projected/4812c628-5f41-4e36-aaf6-8d62e4b6a443-kube-api-access-2rdxt\") pod \"console-operator-584dcb6b54-j5rmn\" (UID: \"4812c628-5f41-4e36-aaf6-8d62e4b6a443\") " pod="openshift-console-operator/console-operator-584dcb6b54-j5rmn" Jul 27 21:04:13.601925 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.601876 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f33ee287-cfc5-436d-93cd-13801c2c522b-tmp\") pod \"insights-operator-5885fdb759-kwqx9\" (UID: \"f33ee287-cfc5-436d-93cd-13801c2c522b\") " pod="openshift-insights/insights-operator-5885fdb759-kwqx9" Jul 27 21:04:13.602034 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.601938 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f33ee287-cfc5-436d-93cd-13801c2c522b-service-ca-bundle\") pod \"insights-operator-5885fdb759-kwqx9\" (UID: \"f33ee287-cfc5-436d-93cd-13801c2c522b\") " pod="openshift-insights/insights-operator-5885fdb759-kwqx9" Jul 27 21:04:13.602034 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.601977 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4812c628-5f41-4e36-aaf6-8d62e4b6a443-serving-cert\") pod \"console-operator-584dcb6b54-j5rmn\" (UID: \"4812c628-5f41-4e36-aaf6-8d62e4b6a443\") " pod="openshift-console-operator/console-operator-584dcb6b54-j5rmn" Jul 27 21:04:13.602034 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.601999 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4812c628-5f41-4e36-aaf6-8d62e4b6a443-trusted-ca\") pod \"console-operator-584dcb6b54-j5rmn\" (UID: \"4812c628-5f41-4e36-aaf6-8d62e4b6a443\") " pod="openshift-console-operator/console-operator-584dcb6b54-j5rmn" Jul 27 21:04:13.602034 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.602033 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f33ee287-cfc5-436d-93cd-13801c2c522b-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-kwqx9\" (UID: \"f33ee287-cfc5-436d-93cd-13801c2c522b\") " pod="openshift-insights/insights-operator-5885fdb759-kwqx9" Jul 27 21:04:13.602227 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.602055 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e066fd54-5796-438f-bdfc-784986a3713c-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-d2494\" (UID: \"e066fd54-5796-438f-bdfc-784986a3713c\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-d2494" Jul 27 21:04:13.602227 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.602075 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4812c628-5f41-4e36-aaf6-8d62e4b6a443-config\") pod \"console-operator-584dcb6b54-j5rmn\" (UID: \"4812c628-5f41-4e36-aaf6-8d62e4b6a443\") " pod="openshift-console-operator/console-operator-584dcb6b54-j5rmn" Jul 27 21:04:13.602227 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.602096 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f33ee287-cfc5-436d-93cd-13801c2c522b-serving-cert\") pod \"insights-operator-5885fdb759-kwqx9\" (UID: \"f33ee287-cfc5-436d-93cd-13801c2c522b\") " pod="openshift-insights/insights-operator-5885fdb759-kwqx9" Jul 27 21:04:13.602227 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.602129 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlc9v\" (UniqueName: \"kubernetes.io/projected/d1562fc4-9850-4084-b2d9-7b388a637f32-kube-api-access-jlc9v\") pod \"cluster-samples-operator-6748489656-zwmht\" (UID: \"d1562fc4-9850-4084-b2d9-7b388a637f32\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-zwmht" Jul 27 21:04:13.602227 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.602174 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5df7w\" (UniqueName: \"kubernetes.io/projected/f33ee287-cfc5-436d-93cd-13801c2c522b-kube-api-access-5df7w\") pod \"insights-operator-5885fdb759-kwqx9\" (UID: \"f33ee287-cfc5-436d-93cd-13801c2c522b\") " pod="openshift-insights/insights-operator-5885fdb759-kwqx9" Jul 27 21:04:13.602227 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.602211 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d1562fc4-9850-4084-b2d9-7b388a637f32-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-zwmht\" (UID: \"d1562fc4-9850-4084-b2d9-7b388a637f32\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-zwmht" Jul 27 21:04:13.602227 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.602230 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f33ee287-cfc5-436d-93cd-13801c2c522b-snapshots\") pod \"insights-operator-5885fdb759-kwqx9\" (UID: \"f33ee287-cfc5-436d-93cd-13801c2c522b\") " pod="openshift-insights/insights-operator-5885fdb759-kwqx9" Jul 27 21:04:13.602588 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.602234 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f33ee287-cfc5-436d-93cd-13801c2c522b-tmp\") pod \"insights-operator-5885fdb759-kwqx9\" (UID: \"f33ee287-cfc5-436d-93cd-13801c2c522b\") " pod="openshift-insights/insights-operator-5885fdb759-kwqx9" Jul 27 21:04:13.602920 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.602808 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f33ee287-cfc5-436d-93cd-13801c2c522b-service-ca-bundle\") pod \"insights-operator-5885fdb759-kwqx9\" (UID: \"f33ee287-cfc5-436d-93cd-13801c2c522b\") " pod="openshift-insights/insights-operator-5885fdb759-kwqx9" Jul 27 21:04:13.602920 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.602821 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f33ee287-cfc5-436d-93cd-13801c2c522b-snapshots\") pod \"insights-operator-5885fdb759-kwqx9\" (UID: \"f33ee287-cfc5-436d-93cd-13801c2c522b\") " pod="openshift-insights/insights-operator-5885fdb759-kwqx9" Jul 27 21:04:13.602920 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:13.602892 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 27 21:04:13.603196 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:13.603028 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1562fc4-9850-4084-b2d9-7b388a637f32-samples-operator-tls podName:d1562fc4-9850-4084-b2d9-7b388a637f32 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:14.103008477 +0000 UTC m=+46.234624681 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d1562fc4-9850-4084-b2d9-7b388a637f32-samples-operator-tls") pod "cluster-samples-operator-6748489656-zwmht" (UID: "d1562fc4-9850-4084-b2d9-7b388a637f32") : secret "samples-operator-tls" not found Jul 27 21:04:13.603414 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:04:13.603380 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce9b0c11_99b0_4de1_bd3b_b5661465bd95.slice/crio-a91ceffac2f98a7e55aecbb922052aa3c6dcb80b5eed0642d3cef58aaa54d3b2 WatchSource:0}: Error finding container a91ceffac2f98a7e55aecbb922052aa3c6dcb80b5eed0642d3cef58aaa54d3b2: Status 404 returned error can't find the container with id a91ceffac2f98a7e55aecbb922052aa3c6dcb80b5eed0642d3cef58aaa54d3b2 Jul 27 21:04:13.603810 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.603790 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f33ee287-cfc5-436d-93cd-13801c2c522b-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-kwqx9\" (UID: \"f33ee287-cfc5-436d-93cd-13801c2c522b\") " pod="openshift-insights/insights-operator-5885fdb759-kwqx9" Jul 27 21:04:13.604763 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.604739 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-9qzp5" Jul 27 21:04:13.604955 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.604934 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f33ee287-cfc5-436d-93cd-13801c2c522b-serving-cert\") pod \"insights-operator-5885fdb759-kwqx9\" (UID: \"f33ee287-cfc5-436d-93cd-13801c2c522b\") " pod="openshift-insights/insights-operator-5885fdb759-kwqx9" Jul 27 21:04:13.619163 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.619139 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5df7w\" (UniqueName: \"kubernetes.io/projected/f33ee287-cfc5-436d-93cd-13801c2c522b-kube-api-access-5df7w\") pod \"insights-operator-5885fdb759-kwqx9\" (UID: \"f33ee287-cfc5-436d-93cd-13801c2c522b\") " pod="openshift-insights/insights-operator-5885fdb759-kwqx9" Jul 27 21:04:13.620944 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.620924 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlc9v\" (UniqueName: \"kubernetes.io/projected/d1562fc4-9850-4084-b2d9-7b388a637f32-kube-api-access-jlc9v\") pod \"cluster-samples-operator-6748489656-zwmht\" (UID: \"d1562fc4-9850-4084-b2d9-7b388a637f32\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-zwmht" Jul 27 21:04:13.633689 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.633660 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-trwhz"] Jul 27 21:04:13.636668 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:04:13.636624 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79a93222_a9cb_4baf_8113_8e46390ff478.slice/crio-7198b5439a63c29aa33a0f0edd68778e2503f99595d614325a244630af052400 WatchSource:0}: Error finding container 7198b5439a63c29aa33a0f0edd68778e2503f99595d614325a244630af052400: Status 404 returned error can't find the container with id 7198b5439a63c29aa33a0f0edd68778e2503f99595d614325a244630af052400 Jul 27 21:04:13.699962 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.699927 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-kwqx9" Jul 27 21:04:13.703266 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.703230 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4812c628-5f41-4e36-aaf6-8d62e4b6a443-serving-cert\") pod \"console-operator-584dcb6b54-j5rmn\" (UID: \"4812c628-5f41-4e36-aaf6-8d62e4b6a443\") " pod="openshift-console-operator/console-operator-584dcb6b54-j5rmn" Jul 27 21:04:13.703415 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.703277 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4812c628-5f41-4e36-aaf6-8d62e4b6a443-trusted-ca\") pod \"console-operator-584dcb6b54-j5rmn\" (UID: \"4812c628-5f41-4e36-aaf6-8d62e4b6a443\") " pod="openshift-console-operator/console-operator-584dcb6b54-j5rmn" Jul 27 21:04:13.703415 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.703323 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e066fd54-5796-438f-bdfc-784986a3713c-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-d2494\" (UID: \"e066fd54-5796-438f-bdfc-784986a3713c\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-d2494" Jul 27 21:04:13.703415 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.703343 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4812c628-5f41-4e36-aaf6-8d62e4b6a443-config\") pod \"console-operator-584dcb6b54-j5rmn\" (UID: \"4812c628-5f41-4e36-aaf6-8d62e4b6a443\") " pod="openshift-console-operator/console-operator-584dcb6b54-j5rmn" Jul 27 21:04:13.703627 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.703430 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e066fd54-5796-438f-bdfc-784986a3713c-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-d2494\" (UID: \"e066fd54-5796-438f-bdfc-784986a3713c\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-d2494" Jul 27 21:04:13.703627 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.703461 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rdxt\" (UniqueName: \"kubernetes.io/projected/4812c628-5f41-4e36-aaf6-8d62e4b6a443-kube-api-access-2rdxt\") pod \"console-operator-584dcb6b54-j5rmn\" (UID: \"4812c628-5f41-4e36-aaf6-8d62e4b6a443\") " pod="openshift-console-operator/console-operator-584dcb6b54-j5rmn" Jul 27 21:04:13.704140 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:13.703842 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 27 21:04:13.704140 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:13.703922 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e066fd54-5796-438f-bdfc-784986a3713c-networking-console-plugin-cert podName:e066fd54-5796-438f-bdfc-784986a3713c nodeName:}" failed. No retries permitted until 2026-07-27 21:04:14.203899043 +0000 UTC m=+46.335515296 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e066fd54-5796-438f-bdfc-784986a3713c-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-d2494" (UID: "e066fd54-5796-438f-bdfc-784986a3713c") : secret "networking-console-plugin-cert" not found Jul 27 21:04:13.704525 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.704497 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4812c628-5f41-4e36-aaf6-8d62e4b6a443-config\") pod \"console-operator-584dcb6b54-j5rmn\" (UID: \"4812c628-5f41-4e36-aaf6-8d62e4b6a443\") " pod="openshift-console-operator/console-operator-584dcb6b54-j5rmn" Jul 27 21:04:13.704692 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.704666 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4812c628-5f41-4e36-aaf6-8d62e4b6a443-trusted-ca\") pod \"console-operator-584dcb6b54-j5rmn\" (UID: \"4812c628-5f41-4e36-aaf6-8d62e4b6a443\") " pod="openshift-console-operator/console-operator-584dcb6b54-j5rmn" Jul 27 21:04:13.704791 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.704748 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e066fd54-5796-438f-bdfc-784986a3713c-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-d2494\" (UID: \"e066fd54-5796-438f-bdfc-784986a3713c\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-d2494" Jul 27 21:04:13.706982 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.706956 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4812c628-5f41-4e36-aaf6-8d62e4b6a443-serving-cert\") pod \"console-operator-584dcb6b54-j5rmn\" (UID: \"4812c628-5f41-4e36-aaf6-8d62e4b6a443\") " pod="openshift-console-operator/console-operator-584dcb6b54-j5rmn" Jul 27 21:04:13.712284 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.712261 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rdxt\" (UniqueName: \"kubernetes.io/projected/4812c628-5f41-4e36-aaf6-8d62e4b6a443-kube-api-access-2rdxt\") pod \"console-operator-584dcb6b54-j5rmn\" (UID: \"4812c628-5f41-4e36-aaf6-8d62e4b6a443\") " pod="openshift-console-operator/console-operator-584dcb6b54-j5rmn" Jul 27 21:04:13.742236 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.742206 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-j5rmn" Jul 27 21:04:13.746249 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.746214 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-9rp9b"] Jul 27 21:04:13.749746 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:04:13.749714 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2e7d196_079c_4bcd_aaf0_affbae8a0594.slice/crio-48163992b1853755c11e10b958309e990a76d87191f3b8fb52eee3ee4a5f22e2 WatchSource:0}: Error finding container 48163992b1853755c11e10b958309e990a76d87191f3b8fb52eee3ee4a5f22e2: Status 404 returned error can't find the container with id 48163992b1853755c11e10b958309e990a76d87191f3b8fb52eee3ee4a5f22e2 Jul 27 21:04:13.760248 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.760219 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-9qzp5"] Jul 27 21:04:13.763220 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:04:13.763192 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4fc83fe_7a79_4461_9d98_c93af693d74f.slice/crio-2b4d8fbb735ebfd7eba2b1e8235e1e83ed9f8d3a7b7a8be903b22892808874f3 WatchSource:0}: Error finding container 2b4d8fbb735ebfd7eba2b1e8235e1e83ed9f8d3a7b7a8be903b22892808874f3: Status 404 returned error can't find the container with id 2b4d8fbb735ebfd7eba2b1e8235e1e83ed9f8d3a7b7a8be903b22892808874f3 Jul 27 21:04:13.870322 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.870285 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-kwqx9"] Jul 27 21:04:13.891150 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:04:13.891065 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf33ee287_cfc5_436d_93cd_13801c2c522b.slice/crio-392160907fc5b8207f3d0ca820f44187de74562569ca6c4f7549a3e8716f4c98 WatchSource:0}: Error finding container 392160907fc5b8207f3d0ca820f44187de74562569ca6c4f7549a3e8716f4c98: Status 404 returned error can't find the container with id 392160907fc5b8207f3d0ca820f44187de74562569ca6c4f7549a3e8716f4c98 Jul 27 21:04:13.897295 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.897264 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-j5rmn"] Jul 27 21:04:13.902343 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:04:13.902304 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4812c628_5f41_4e36_aaf6_8d62e4b6a443.slice/crio-0eac31be102876655773242b8cf19202911992b87cc57c265c7a95d1d9ba2f89 WatchSource:0}: Error finding container 0eac31be102876655773242b8cf19202911992b87cc57c265c7a95d1d9ba2f89: Status 404 returned error can't find the container with id 0eac31be102876655773242b8cf19202911992b87cc57c265c7a95d1d9ba2f89 Jul 27 21:04:13.904208 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.904186 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86e4b5a1-4521-43b3-9bf8-c42ec92d21f8-metrics-certs\") pod \"router-default-64c559777f-75n27\" (UID: \"86e4b5a1-4521-43b3-9bf8-c42ec92d21f8\") " pod="openshift-ingress/router-default-64c559777f-75n27" Jul 27 21:04:13.904291 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:13.904231 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86e4b5a1-4521-43b3-9bf8-c42ec92d21f8-service-ca-bundle\") pod \"router-default-64c559777f-75n27\" (UID: \"86e4b5a1-4521-43b3-9bf8-c42ec92d21f8\") " pod="openshift-ingress/router-default-64c559777f-75n27" Jul 27 21:04:13.904351 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:13.904288 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 27 21:04:13.904351 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:13.904346 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86e4b5a1-4521-43b3-9bf8-c42ec92d21f8-metrics-certs podName:86e4b5a1-4521-43b3-9bf8-c42ec92d21f8 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:14.904327178 +0000 UTC m=+47.035943394 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86e4b5a1-4521-43b3-9bf8-c42ec92d21f8-metrics-certs") pod "router-default-64c559777f-75n27" (UID: "86e4b5a1-4521-43b3-9bf8-c42ec92d21f8") : secret "router-metrics-certs-default" not found Jul 27 21:04:13.904505 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:13.904396 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/86e4b5a1-4521-43b3-9bf8-c42ec92d21f8-service-ca-bundle podName:86e4b5a1-4521-43b3-9bf8-c42ec92d21f8 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:14.904381076 +0000 UTC m=+47.035997277 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/86e4b5a1-4521-43b3-9bf8-c42ec92d21f8-service-ca-bundle") pod "router-default-64c559777f-75n27" (UID: "86e4b5a1-4521-43b3-9bf8-c42ec92d21f8") : configmap references non-existent config key: service-ca.crt Jul 27 21:04:14.004754 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:14.004710 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/91a29cde-048a-4e5e-a804-5854f86081ed-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-dm69z\" (UID: \"91a29cde-048a-4e5e-a804-5854f86081ed\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-dm69z" Jul 27 21:04:14.004927 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:14.004862 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 27 21:04:14.004927 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:14.004926 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91a29cde-048a-4e5e-a804-5854f86081ed-cluster-monitoring-operator-tls podName:91a29cde-048a-4e5e-a804-5854f86081ed nodeName:}" failed. No retries permitted until 2026-07-27 21:04:15.004909197 +0000 UTC m=+47.136525405 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/91a29cde-048a-4e5e-a804-5854f86081ed-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-dm69z" (UID: "91a29cde-048a-4e5e-a804-5854f86081ed") : secret "cluster-monitoring-operator-tls" not found Jul 27 21:04:14.105198 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:14.105162 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d1562fc4-9850-4084-b2d9-7b388a637f32-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-zwmht\" (UID: \"d1562fc4-9850-4084-b2d9-7b388a637f32\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-zwmht" Jul 27 21:04:14.105345 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:14.105310 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 27 21:04:14.105399 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:14.105377 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1562fc4-9850-4084-b2d9-7b388a637f32-samples-operator-tls podName:d1562fc4-9850-4084-b2d9-7b388a637f32 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:15.105362826 +0000 UTC m=+47.236979030 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d1562fc4-9850-4084-b2d9-7b388a637f32-samples-operator-tls") pod "cluster-samples-operator-6748489656-zwmht" (UID: "d1562fc4-9850-4084-b2d9-7b388a637f32") : secret "samples-operator-tls" not found Jul 27 21:04:14.206122 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:14.206032 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e066fd54-5796-438f-bdfc-784986a3713c-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-d2494\" (UID: \"e066fd54-5796-438f-bdfc-784986a3713c\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-d2494" Jul 27 21:04:14.206565 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:14.206170 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 27 21:04:14.206565 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:14.206227 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e066fd54-5796-438f-bdfc-784986a3713c-networking-console-plugin-cert podName:e066fd54-5796-438f-bdfc-784986a3713c nodeName:}" failed. No retries permitted until 2026-07-27 21:04:15.206213351 +0000 UTC m=+47.337829552 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e066fd54-5796-438f-bdfc-784986a3713c-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-d2494" (UID: "e066fd54-5796-438f-bdfc-784986a3713c") : secret "networking-console-plugin-cert" not found Jul 27 21:04:14.580301 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:14.580218 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-j5rmn" event={"ID":"4812c628-5f41-4e36-aaf6-8d62e4b6a443","Type":"ContainerStarted","Data":"0eac31be102876655773242b8cf19202911992b87cc57c265c7a95d1d9ba2f89"} Jul 27 21:04:14.582838 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:14.582751 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-kwqx9" event={"ID":"f33ee287-cfc5-436d-93cd-13801c2c522b","Type":"ContainerStarted","Data":"392160907fc5b8207f3d0ca820f44187de74562569ca6c4f7549a3e8716f4c98"} Jul 27 21:04:14.585134 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:14.585004 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-gdgs7" event={"ID":"ce9b0c11-99b0-4de1-bd3b-b5661465bd95","Type":"ContainerStarted","Data":"a91ceffac2f98a7e55aecbb922052aa3c6dcb80b5eed0642d3cef58aaa54d3b2"} Jul 27 21:04:14.587898 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:14.587818 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-9qzp5" event={"ID":"a4fc83fe-7a79-4461-9d98-c93af693d74f","Type":"ContainerStarted","Data":"2b4d8fbb735ebfd7eba2b1e8235e1e83ed9f8d3a7b7a8be903b22892808874f3"} Jul 27 21:04:14.590898 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:14.590821 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-9rp9b" event={"ID":"b2e7d196-079c-4bcd-aaf0-affbae8a0594","Type":"ContainerStarted","Data":"48163992b1853755c11e10b958309e990a76d87191f3b8fb52eee3ee4a5f22e2"} Jul 27 21:04:14.592382 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:14.592316 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-trwhz" event={"ID":"79a93222-a9cb-4baf-8113-8e46390ff478","Type":"ContainerStarted","Data":"7198b5439a63c29aa33a0f0edd68778e2503f99595d614325a244630af052400"} Jul 27 21:04:14.912014 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:14.911917 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86e4b5a1-4521-43b3-9bf8-c42ec92d21f8-service-ca-bundle\") pod \"router-default-64c559777f-75n27\" (UID: \"86e4b5a1-4521-43b3-9bf8-c42ec92d21f8\") " pod="openshift-ingress/router-default-64c559777f-75n27" Jul 27 21:04:14.912192 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:14.912104 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86e4b5a1-4521-43b3-9bf8-c42ec92d21f8-metrics-certs\") pod \"router-default-64c559777f-75n27\" (UID: \"86e4b5a1-4521-43b3-9bf8-c42ec92d21f8\") " pod="openshift-ingress/router-default-64c559777f-75n27" Jul 27 21:04:14.912273 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:14.912254 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 27 21:04:14.912328 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:14.912322 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86e4b5a1-4521-43b3-9bf8-c42ec92d21f8-metrics-certs podName:86e4b5a1-4521-43b3-9bf8-c42ec92d21f8 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:16.912303615 +0000 UTC m=+49.043919885 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86e4b5a1-4521-43b3-9bf8-c42ec92d21f8-metrics-certs") pod "router-default-64c559777f-75n27" (UID: "86e4b5a1-4521-43b3-9bf8-c42ec92d21f8") : secret "router-metrics-certs-default" not found Jul 27 21:04:14.912731 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:14.912690 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/86e4b5a1-4521-43b3-9bf8-c42ec92d21f8-service-ca-bundle podName:86e4b5a1-4521-43b3-9bf8-c42ec92d21f8 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:16.912673015 +0000 UTC m=+49.044289235 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/86e4b5a1-4521-43b3-9bf8-c42ec92d21f8-service-ca-bundle") pod "router-default-64c559777f-75n27" (UID: "86e4b5a1-4521-43b3-9bf8-c42ec92d21f8") : configmap references non-existent config key: service-ca.crt Jul 27 21:04:15.013386 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:15.012832 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/91a29cde-048a-4e5e-a804-5854f86081ed-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-dm69z\" (UID: \"91a29cde-048a-4e5e-a804-5854f86081ed\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-dm69z" Jul 27 21:04:15.013386 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:15.012983 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 27 21:04:15.013386 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:15.013046 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91a29cde-048a-4e5e-a804-5854f86081ed-cluster-monitoring-operator-tls podName:91a29cde-048a-4e5e-a804-5854f86081ed nodeName:}" failed. No retries permitted until 2026-07-27 21:04:17.013027163 +0000 UTC m=+49.144643388 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/91a29cde-048a-4e5e-a804-5854f86081ed-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-dm69z" (UID: "91a29cde-048a-4e5e-a804-5854f86081ed") : secret "cluster-monitoring-operator-tls" not found Jul 27 21:04:15.114878 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:15.114196 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d1562fc4-9850-4084-b2d9-7b388a637f32-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-zwmht\" (UID: \"d1562fc4-9850-4084-b2d9-7b388a637f32\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-zwmht" Jul 27 21:04:15.114878 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:15.114481 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 27 21:04:15.114878 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:15.114545 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1562fc4-9850-4084-b2d9-7b388a637f32-samples-operator-tls podName:d1562fc4-9850-4084-b2d9-7b388a637f32 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:17.114524331 +0000 UTC m=+49.246140538 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d1562fc4-9850-4084-b2d9-7b388a637f32-samples-operator-tls") pod "cluster-samples-operator-6748489656-zwmht" (UID: "d1562fc4-9850-4084-b2d9-7b388a637f32") : secret "samples-operator-tls" not found Jul 27 21:04:15.215550 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:15.214893 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e066fd54-5796-438f-bdfc-784986a3713c-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-d2494\" (UID: \"e066fd54-5796-438f-bdfc-784986a3713c\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-d2494" Jul 27 21:04:15.215550 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:15.215095 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 27 21:04:15.215550 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:15.215160 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e066fd54-5796-438f-bdfc-784986a3713c-networking-console-plugin-cert podName:e066fd54-5796-438f-bdfc-784986a3713c nodeName:}" failed. No retries permitted until 2026-07-27 21:04:17.215140561 +0000 UTC m=+49.346756765 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e066fd54-5796-438f-bdfc-784986a3713c-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-d2494" (UID: "e066fd54-5796-438f-bdfc-784986a3713c") : secret "networking-console-plugin-cert" not found Jul 27 21:04:16.933242 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:16.933202 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86e4b5a1-4521-43b3-9bf8-c42ec92d21f8-metrics-certs\") pod \"router-default-64c559777f-75n27\" (UID: \"86e4b5a1-4521-43b3-9bf8-c42ec92d21f8\") " pod="openshift-ingress/router-default-64c559777f-75n27" Jul 27 21:04:16.933740 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:16.933325 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 27 21:04:16.933740 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:16.933342 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86e4b5a1-4521-43b3-9bf8-c42ec92d21f8-service-ca-bundle\") pod \"router-default-64c559777f-75n27\" (UID: \"86e4b5a1-4521-43b3-9bf8-c42ec92d21f8\") " pod="openshift-ingress/router-default-64c559777f-75n27" Jul 27 21:04:16.933740 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:16.933395 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86e4b5a1-4521-43b3-9bf8-c42ec92d21f8-metrics-certs podName:86e4b5a1-4521-43b3-9bf8-c42ec92d21f8 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:20.933373663 +0000 UTC m=+53.064989888 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86e4b5a1-4521-43b3-9bf8-c42ec92d21f8-metrics-certs") pod "router-default-64c559777f-75n27" (UID: "86e4b5a1-4521-43b3-9bf8-c42ec92d21f8") : secret "router-metrics-certs-default" not found Jul 27 21:04:16.933740 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:16.933517 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/86e4b5a1-4521-43b3-9bf8-c42ec92d21f8-service-ca-bundle podName:86e4b5a1-4521-43b3-9bf8-c42ec92d21f8 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:20.933479342 +0000 UTC m=+53.065095558 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/86e4b5a1-4521-43b3-9bf8-c42ec92d21f8-service-ca-bundle") pod "router-default-64c559777f-75n27" (UID: "86e4b5a1-4521-43b3-9bf8-c42ec92d21f8") : configmap references non-existent config key: service-ca.crt Jul 27 21:04:17.034876 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:17.034836 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/91a29cde-048a-4e5e-a804-5854f86081ed-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-dm69z\" (UID: \"91a29cde-048a-4e5e-a804-5854f86081ed\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-dm69z" Jul 27 21:04:17.035144 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:17.035125 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 27 21:04:17.035211 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:17.035202 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91a29cde-048a-4e5e-a804-5854f86081ed-cluster-monitoring-operator-tls podName:91a29cde-048a-4e5e-a804-5854f86081ed nodeName:}" failed. No retries permitted until 2026-07-27 21:04:21.035179054 +0000 UTC m=+53.166795273 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/91a29cde-048a-4e5e-a804-5854f86081ed-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-dm69z" (UID: "91a29cde-048a-4e5e-a804-5854f86081ed") : secret "cluster-monitoring-operator-tls" not found Jul 27 21:04:17.135467 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:17.135428 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d1562fc4-9850-4084-b2d9-7b388a637f32-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-zwmht\" (UID: \"d1562fc4-9850-4084-b2d9-7b388a637f32\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-zwmht" Jul 27 21:04:17.135648 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:17.135555 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 27 21:04:17.135648 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:17.135642 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1562fc4-9850-4084-b2d9-7b388a637f32-samples-operator-tls podName:d1562fc4-9850-4084-b2d9-7b388a637f32 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:21.13562215 +0000 UTC m=+53.267238365 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d1562fc4-9850-4084-b2d9-7b388a637f32-samples-operator-tls") pod "cluster-samples-operator-6748489656-zwmht" (UID: "d1562fc4-9850-4084-b2d9-7b388a637f32") : secret "samples-operator-tls" not found Jul 27 21:04:17.236617 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:17.236581 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e066fd54-5796-438f-bdfc-784986a3713c-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-d2494\" (UID: \"e066fd54-5796-438f-bdfc-784986a3713c\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-d2494" Jul 27 21:04:17.236778 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:17.236690 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 27 21:04:17.236778 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:17.236769 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e066fd54-5796-438f-bdfc-784986a3713c-networking-console-plugin-cert podName:e066fd54-5796-438f-bdfc-784986a3713c nodeName:}" failed. No retries permitted until 2026-07-27 21:04:21.236750193 +0000 UTC m=+53.368366408 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e066fd54-5796-438f-bdfc-784986a3713c-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-d2494" (UID: "e066fd54-5796-438f-bdfc-784986a3713c") : secret "networking-console-plugin-cert" not found Jul 27 21:04:20.164199 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:20.164165 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/932ad643-a059-44a4-90f8-41016fc1e5c0-metrics-tls\") pod \"dns-default-ppf68\" (UID: \"932ad643-a059-44a4-90f8-41016fc1e5c0\") " pod="openshift-dns/dns-default-ppf68" Jul 27 21:04:20.164625 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:20.164213 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eb9ca170-7ea1-45de-aca4-54b4731ed82d-registry-tls\") pod \"image-registry-75fb88d74f-jhdnw\" (UID: \"eb9ca170-7ea1-45de-aca4-54b4731ed82d\") " pod="openshift-image-registry/image-registry-75fb88d74f-jhdnw" Jul 27 21:04:20.164625 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:20.164312 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 27 21:04:20.164625 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:20.164322 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75fb88d74f-jhdnw: secret "image-registry-tls" not found Jul 27 21:04:20.164625 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:20.164328 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 21:04:20.164625 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:20.164367 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb9ca170-7ea1-45de-aca4-54b4731ed82d-registry-tls podName:eb9ca170-7ea1-45de-aca4-54b4731ed82d nodeName:}" failed. No retries permitted until 2026-07-27 21:04:36.164354742 +0000 UTC m=+68.295970943 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/eb9ca170-7ea1-45de-aca4-54b4731ed82d-registry-tls") pod "image-registry-75fb88d74f-jhdnw" (UID: "eb9ca170-7ea1-45de-aca4-54b4731ed82d") : secret "image-registry-tls" not found Jul 27 21:04:20.164625 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:20.164467 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/932ad643-a059-44a4-90f8-41016fc1e5c0-metrics-tls podName:932ad643-a059-44a4-90f8-41016fc1e5c0 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:36.164446192 +0000 UTC m=+68.296062406 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/932ad643-a059-44a4-90f8-41016fc1e5c0-metrics-tls") pod "dns-default-ppf68" (UID: "932ad643-a059-44a4-90f8-41016fc1e5c0") : secret "dns-default-metrics-tls" not found Jul 27 21:04:20.613844 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:20.613269 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-9rp9b" event={"ID":"b2e7d196-079c-4bcd-aaf0-affbae8a0594","Type":"ContainerStarted","Data":"eaa0a079061ba39109b685264deb4a8e6b352bfff4e5eb755d4edf013495a517"} Jul 27 21:04:20.615481 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:20.615448 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-trwhz" event={"ID":"79a93222-a9cb-4baf-8113-8e46390ff478","Type":"ContainerStarted","Data":"6e82f7d7696e085ebc94a01f0bf6b61edc6eedbb07708387ae2eb25dd8c1cb37"} Jul 27 21:04:20.617518 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:20.617492 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-j5rmn_4812c628-5f41-4e36-aaf6-8d62e4b6a443/console-operator/0.log" Jul 27 21:04:20.617673 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:20.617537 2577 generic.go:356] "Generic (PLEG): container finished" podID="4812c628-5f41-4e36-aaf6-8d62e4b6a443" containerID="def1207e36a87b964c312244c0c444778e753e25c04a4f5e4b1dd8ff4fce5f94" exitCode=255 Jul 27 21:04:20.617673 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:20.617631 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-j5rmn" event={"ID":"4812c628-5f41-4e36-aaf6-8d62e4b6a443","Type":"ContainerDied","Data":"def1207e36a87b964c312244c0c444778e753e25c04a4f5e4b1dd8ff4fce5f94"} Jul 27 21:04:20.617876 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:20.617856 2577 scope.go:117] "RemoveContainer" containerID="def1207e36a87b964c312244c0c444778e753e25c04a4f5e4b1dd8ff4fce5f94" Jul 27 21:04:20.619322 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:20.619296 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-kwqx9" event={"ID":"f33ee287-cfc5-436d-93cd-13801c2c522b","Type":"ContainerStarted","Data":"0ab0db102c864826e0a3119779c9d5030b1d639080af9576478ebb0fc79c30e6"} Jul 27 21:04:20.621607 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:20.621544 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-gdgs7" event={"ID":"ce9b0c11-99b0-4de1-bd3b-b5661465bd95","Type":"ContainerStarted","Data":"d96044c8baeab7e87058f30bcd1369ab4719a08d2733345deb9f7b700b48c5ee"} Jul 27 21:04:20.623286 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:20.623258 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-9qzp5" event={"ID":"a4fc83fe-7a79-4461-9d98-c93af693d74f","Type":"ContainerStarted","Data":"7b9a24da0c81bf07fed41bc139d34c7f34c16e00332dd8c4b0ec7ec9a682e82a"} Jul 27 21:04:20.630156 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:20.630099 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-9rp9b" podStartSLOduration=1.299007253 podStartE2EDuration="7.630080137s" totalImagesPullingTime="6.331072884s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:04:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:20.62915174 +0000 UTC m=+52.760767976" watchObservedRunningTime="2026-07-27 21:04:20.630080137 +0000 UTC m=+52.761696358" Jul 27 21:04:20.660554 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:20.660344 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-trwhz" podStartSLOduration=1.216835496 podStartE2EDuration="7.660319836s" totalImagesPullingTime="6.44348434s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:04:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:20.648692855 +0000 UTC m=+52.780309070" watchObservedRunningTime="2026-07-27 21:04:20.660319836 +0000 UTC m=+52.791936060" Jul 27 21:04:20.675472 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:20.675415 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-kwqx9" podStartSLOduration=1.4797781159999999 podStartE2EDuration="7.675393171s" totalImagesPullingTime="6.195615055s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:04:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:20.673866662 +0000 UTC m=+52.805482886" watchObservedRunningTime="2026-07-27 21:04:20.675393171 +0000 UTC m=+52.807009396" Jul 27 21:04:20.693775 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:20.693483 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-gdgs7" podStartSLOduration=1.220943074 podStartE2EDuration="7.693465593s" totalImagesPullingTime="6.472522519s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:04:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:20.692525626 +0000 UTC m=+52.824141851" watchObservedRunningTime="2026-07-27 21:04:20.693465593 +0000 UTC m=+52.825081817" Jul 27 21:04:20.710790 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:20.710717 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-9qzp5" podStartSLOduration=1.393442314 podStartE2EDuration="7.710702167s" totalImagesPullingTime="6.317259853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:04:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:20.710342966 +0000 UTC m=+52.841959191" watchObservedRunningTime="2026-07-27 21:04:20.710702167 +0000 UTC m=+52.842318393" Jul 27 21:04:20.971900 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:20.971854 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86e4b5a1-4521-43b3-9bf8-c42ec92d21f8-metrics-certs\") pod \"router-default-64c559777f-75n27\" (UID: \"86e4b5a1-4521-43b3-9bf8-c42ec92d21f8\") " pod="openshift-ingress/router-default-64c559777f-75n27" Jul 27 21:04:20.972107 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:20.971947 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86e4b5a1-4521-43b3-9bf8-c42ec92d21f8-service-ca-bundle\") pod \"router-default-64c559777f-75n27\" (UID: \"86e4b5a1-4521-43b3-9bf8-c42ec92d21f8\") " pod="openshift-ingress/router-default-64c559777f-75n27" Jul 27 21:04:20.972107 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:20.972001 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 27 21:04:20.972107 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:20.972088 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86e4b5a1-4521-43b3-9bf8-c42ec92d21f8-metrics-certs podName:86e4b5a1-4521-43b3-9bf8-c42ec92d21f8 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:28.972064022 +0000 UTC m=+61.103680239 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86e4b5a1-4521-43b3-9bf8-c42ec92d21f8-metrics-certs") pod "router-default-64c559777f-75n27" (UID: "86e4b5a1-4521-43b3-9bf8-c42ec92d21f8") : secret "router-metrics-certs-default" not found Jul 27 21:04:20.972284 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:20.972127 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/86e4b5a1-4521-43b3-9bf8-c42ec92d21f8-service-ca-bundle podName:86e4b5a1-4521-43b3-9bf8-c42ec92d21f8 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:28.972106721 +0000 UTC m=+61.103722924 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/86e4b5a1-4521-43b3-9bf8-c42ec92d21f8-service-ca-bundle") pod "router-default-64c559777f-75n27" (UID: "86e4b5a1-4521-43b3-9bf8-c42ec92d21f8") : configmap references non-existent config key: service-ca.crt Jul 27 21:04:21.072886 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:21.072843 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/91a29cde-048a-4e5e-a804-5854f86081ed-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-dm69z\" (UID: \"91a29cde-048a-4e5e-a804-5854f86081ed\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-dm69z" Jul 27 21:04:21.073045 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:21.073018 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 27 21:04:21.073122 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:21.073110 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91a29cde-048a-4e5e-a804-5854f86081ed-cluster-monitoring-operator-tls podName:91a29cde-048a-4e5e-a804-5854f86081ed nodeName:}" failed. No retries permitted until 2026-07-27 21:04:29.073087312 +0000 UTC m=+61.204703528 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/91a29cde-048a-4e5e-a804-5854f86081ed-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-dm69z" (UID: "91a29cde-048a-4e5e-a804-5854f86081ed") : secret "cluster-monitoring-operator-tls" not found Jul 27 21:04:21.173332 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:21.173288 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d1562fc4-9850-4084-b2d9-7b388a637f32-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-zwmht\" (UID: \"d1562fc4-9850-4084-b2d9-7b388a637f32\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-zwmht" Jul 27 21:04:21.173813 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:21.173455 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 27 21:04:21.173813 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:21.173533 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1562fc4-9850-4084-b2d9-7b388a637f32-samples-operator-tls podName:d1562fc4-9850-4084-b2d9-7b388a637f32 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:29.173510582 +0000 UTC m=+61.305126785 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d1562fc4-9850-4084-b2d9-7b388a637f32-samples-operator-tls") pod "cluster-samples-operator-6748489656-zwmht" (UID: "d1562fc4-9850-4084-b2d9-7b388a637f32") : secret "samples-operator-tls" not found Jul 27 21:04:21.274783 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:21.274154 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e066fd54-5796-438f-bdfc-784986a3713c-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-d2494\" (UID: \"e066fd54-5796-438f-bdfc-784986a3713c\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-d2494" Jul 27 21:04:21.274783 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:21.274481 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 27 21:04:21.274783 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:21.274552 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e066fd54-5796-438f-bdfc-784986a3713c-networking-console-plugin-cert podName:e066fd54-5796-438f-bdfc-784986a3713c nodeName:}" failed. No retries permitted until 2026-07-27 21:04:29.274533294 +0000 UTC m=+61.406149518 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e066fd54-5796-438f-bdfc-784986a3713c-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-d2494" (UID: "e066fd54-5796-438f-bdfc-784986a3713c") : secret "networking-console-plugin-cert" not found Jul 27 21:04:21.628210 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:21.628126 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-j5rmn_4812c628-5f41-4e36-aaf6-8d62e4b6a443/console-operator/1.log" Jul 27 21:04:21.628550 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:21.628533 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-j5rmn_4812c628-5f41-4e36-aaf6-8d62e4b6a443/console-operator/0.log" Jul 27 21:04:21.628643 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:21.628589 2577 generic.go:356] "Generic (PLEG): container finished" podID="4812c628-5f41-4e36-aaf6-8d62e4b6a443" containerID="010d6f34a18c34bbd4ce69002f8598033b36b045e6606777f010b27ef6844930" exitCode=255 Jul 27 21:04:21.628713 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:21.628688 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-j5rmn" event={"ID":"4812c628-5f41-4e36-aaf6-8d62e4b6a443","Type":"ContainerDied","Data":"010d6f34a18c34bbd4ce69002f8598033b36b045e6606777f010b27ef6844930"} Jul 27 21:04:21.628798 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:21.628786 2577 scope.go:117] "RemoveContainer" containerID="def1207e36a87b964c312244c0c444778e753e25c04a4f5e4b1dd8ff4fce5f94" Jul 27 21:04:21.628930 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:21.628915 2577 scope.go:117] "RemoveContainer" containerID="010d6f34a18c34bbd4ce69002f8598033b36b045e6606777f010b27ef6844930" Jul 27 21:04:21.629118 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:21.629090 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-j5rmn_openshift-console-operator(4812c628-5f41-4e36-aaf6-8d62e4b6a443)\"" pod="openshift-console-operator/console-operator-584dcb6b54-j5rmn" podUID="4812c628-5f41-4e36-aaf6-8d62e4b6a443" Jul 27 21:04:21.932775 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:21.932689 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-wkkhn"] Jul 27 21:04:21.961686 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:21.961655 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wkkhn"] Jul 27 21:04:21.961827 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:21.961780 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wkkhn" Jul 27 21:04:21.964493 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:21.964463 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 27 21:04:21.964630 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:21.964586 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 27 21:04:21.964762 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:21.964749 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-w5dc2\"" Jul 27 21:04:22.083221 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:22.083178 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2abb5699-2333-4c3f-b748-d3a227fbfb9c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wkkhn\" (UID: \"2abb5699-2333-4c3f-b748-d3a227fbfb9c\") " pod="openshift-insights/insights-runtime-extractor-wkkhn" Jul 27 21:04:22.083221 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:22.083222 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2abb5699-2333-4c3f-b748-d3a227fbfb9c-crio-socket\") pod \"insights-runtime-extractor-wkkhn\" (UID: \"2abb5699-2333-4c3f-b748-d3a227fbfb9c\") " pod="openshift-insights/insights-runtime-extractor-wkkhn" Jul 27 21:04:22.083431 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:22.083329 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2abb5699-2333-4c3f-b748-d3a227fbfb9c-data-volume\") pod \"insights-runtime-extractor-wkkhn\" (UID: \"2abb5699-2333-4c3f-b748-d3a227fbfb9c\") " pod="openshift-insights/insights-runtime-extractor-wkkhn" Jul 27 21:04:22.083431 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:22.083373 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh6ht\" (UniqueName: \"kubernetes.io/projected/2abb5699-2333-4c3f-b748-d3a227fbfb9c-kube-api-access-qh6ht\") pod \"insights-runtime-extractor-wkkhn\" (UID: \"2abb5699-2333-4c3f-b748-d3a227fbfb9c\") " pod="openshift-insights/insights-runtime-extractor-wkkhn" Jul 27 21:04:22.083431 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:22.083409 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2abb5699-2333-4c3f-b748-d3a227fbfb9c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wkkhn\" (UID: \"2abb5699-2333-4c3f-b748-d3a227fbfb9c\") " pod="openshift-insights/insights-runtime-extractor-wkkhn" Jul 27 21:04:22.184373 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:22.184269 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh6ht\" (UniqueName: \"kubernetes.io/projected/2abb5699-2333-4c3f-b748-d3a227fbfb9c-kube-api-access-qh6ht\") pod \"insights-runtime-extractor-wkkhn\" (UID: \"2abb5699-2333-4c3f-b748-d3a227fbfb9c\") " pod="openshift-insights/insights-runtime-extractor-wkkhn" Jul 27 21:04:22.184373 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:22.184339 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2abb5699-2333-4c3f-b748-d3a227fbfb9c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wkkhn\" (UID: \"2abb5699-2333-4c3f-b748-d3a227fbfb9c\") " pod="openshift-insights/insights-runtime-extractor-wkkhn" Jul 27 21:04:22.184872 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:22.184470 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2abb5699-2333-4c3f-b748-d3a227fbfb9c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wkkhn\" (UID: \"2abb5699-2333-4c3f-b748-d3a227fbfb9c\") " pod="openshift-insights/insights-runtime-extractor-wkkhn" Jul 27 21:04:22.184872 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:22.184482 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 27 21:04:22.184872 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:22.184502 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2abb5699-2333-4c3f-b748-d3a227fbfb9c-crio-socket\") pod \"insights-runtime-extractor-wkkhn\" (UID: \"2abb5699-2333-4c3f-b748-d3a227fbfb9c\") " pod="openshift-insights/insights-runtime-extractor-wkkhn" Jul 27 21:04:22.184872 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:22.184564 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2abb5699-2333-4c3f-b748-d3a227fbfb9c-kube-rbac-proxy-cm podName:2abb5699-2333-4c3f-b748-d3a227fbfb9c nodeName:}" failed. No retries permitted until 2026-07-27 21:04:22.684542736 +0000 UTC m=+54.816158965 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2abb5699-2333-4c3f-b748-d3a227fbfb9c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wkkhn" (UID: "2abb5699-2333-4c3f-b748-d3a227fbfb9c") : configmap "kube-rbac-proxy" not found Jul 27 21:04:22.184872 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:22.184642 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 21:04:22.184872 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:22.184645 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2abb5699-2333-4c3f-b748-d3a227fbfb9c-data-volume\") pod \"insights-runtime-extractor-wkkhn\" (UID: \"2abb5699-2333-4c3f-b748-d3a227fbfb9c\") " pod="openshift-insights/insights-runtime-extractor-wkkhn" Jul 27 21:04:22.184872 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:22.184694 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2abb5699-2333-4c3f-b748-d3a227fbfb9c-insights-runtime-extractor-tls podName:2abb5699-2333-4c3f-b748-d3a227fbfb9c nodeName:}" failed. No retries permitted until 2026-07-27 21:04:22.684680979 +0000 UTC m=+54.816297179 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2abb5699-2333-4c3f-b748-d3a227fbfb9c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wkkhn" (UID: "2abb5699-2333-4c3f-b748-d3a227fbfb9c") : secret "insights-runtime-extractor-tls" not found Jul 27 21:04:22.184872 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:22.184738 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2abb5699-2333-4c3f-b748-d3a227fbfb9c-crio-socket\") pod \"insights-runtime-extractor-wkkhn\" (UID: \"2abb5699-2333-4c3f-b748-d3a227fbfb9c\") " pod="openshift-insights/insights-runtime-extractor-wkkhn" Jul 27 21:04:22.185130 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:22.184922 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2abb5699-2333-4c3f-b748-d3a227fbfb9c-data-volume\") pod \"insights-runtime-extractor-wkkhn\" (UID: \"2abb5699-2333-4c3f-b748-d3a227fbfb9c\") " pod="openshift-insights/insights-runtime-extractor-wkkhn" Jul 27 21:04:22.196197 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:22.196165 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh6ht\" (UniqueName: \"kubernetes.io/projected/2abb5699-2333-4c3f-b748-d3a227fbfb9c-kube-api-access-qh6ht\") pod \"insights-runtime-extractor-wkkhn\" (UID: \"2abb5699-2333-4c3f-b748-d3a227fbfb9c\") " pod="openshift-insights/insights-runtime-extractor-wkkhn" Jul 27 21:04:22.634096 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:22.634066 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-j5rmn_4812c628-5f41-4e36-aaf6-8d62e4b6a443/console-operator/1.log" Jul 27 21:04:22.634413 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:22.634397 2577 scope.go:117] "RemoveContainer" containerID="010d6f34a18c34bbd4ce69002f8598033b36b045e6606777f010b27ef6844930" Jul 27 21:04:22.634606 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:22.634584 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-j5rmn_openshift-console-operator(4812c628-5f41-4e36-aaf6-8d62e4b6a443)\"" pod="openshift-console-operator/console-operator-584dcb6b54-j5rmn" podUID="4812c628-5f41-4e36-aaf6-8d62e4b6a443" Jul 27 21:04:22.688791 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:22.688754 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2abb5699-2333-4c3f-b748-d3a227fbfb9c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wkkhn\" (UID: \"2abb5699-2333-4c3f-b748-d3a227fbfb9c\") " pod="openshift-insights/insights-runtime-extractor-wkkhn" Jul 27 21:04:22.689070 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:22.689038 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 21:04:22.689213 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:22.689127 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2abb5699-2333-4c3f-b748-d3a227fbfb9c-insights-runtime-extractor-tls podName:2abb5699-2333-4c3f-b748-d3a227fbfb9c nodeName:}" failed. No retries permitted until 2026-07-27 21:04:23.68910416 +0000 UTC m=+55.820720364 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2abb5699-2333-4c3f-b748-d3a227fbfb9c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wkkhn" (UID: "2abb5699-2333-4c3f-b748-d3a227fbfb9c") : secret "insights-runtime-extractor-tls" not found Jul 27 21:04:22.689285 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:22.689263 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2abb5699-2333-4c3f-b748-d3a227fbfb9c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wkkhn\" (UID: \"2abb5699-2333-4c3f-b748-d3a227fbfb9c\") " pod="openshift-insights/insights-runtime-extractor-wkkhn" Jul 27 21:04:22.689347 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:22.689310 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 27 21:04:22.689396 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:22.689364 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2abb5699-2333-4c3f-b748-d3a227fbfb9c-kube-rbac-proxy-cm podName:2abb5699-2333-4c3f-b748-d3a227fbfb9c nodeName:}" failed. No retries permitted until 2026-07-27 21:04:23.689348603 +0000 UTC m=+55.820964808 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2abb5699-2333-4c3f-b748-d3a227fbfb9c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wkkhn" (UID: "2abb5699-2333-4c3f-b748-d3a227fbfb9c") : configmap "kube-rbac-proxy" not found Jul 27 21:04:23.699316 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:23.699255 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2abb5699-2333-4c3f-b748-d3a227fbfb9c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wkkhn\" (UID: \"2abb5699-2333-4c3f-b748-d3a227fbfb9c\") " pod="openshift-insights/insights-runtime-extractor-wkkhn" Jul 27 21:04:23.699754 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:23.699382 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 27 21:04:23.699754 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:23.699427 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2abb5699-2333-4c3f-b748-d3a227fbfb9c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wkkhn\" (UID: \"2abb5699-2333-4c3f-b748-d3a227fbfb9c\") " pod="openshift-insights/insights-runtime-extractor-wkkhn" Jul 27 21:04:23.699754 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:23.699453 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2abb5699-2333-4c3f-b748-d3a227fbfb9c-kube-rbac-proxy-cm podName:2abb5699-2333-4c3f-b748-d3a227fbfb9c nodeName:}" failed. No retries permitted until 2026-07-27 21:04:25.699433374 +0000 UTC m=+57.831049583 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2abb5699-2333-4c3f-b748-d3a227fbfb9c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wkkhn" (UID: "2abb5699-2333-4c3f-b748-d3a227fbfb9c") : configmap "kube-rbac-proxy" not found Jul 27 21:04:23.699754 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:23.699515 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 21:04:23.699754 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:23.699557 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2abb5699-2333-4c3f-b748-d3a227fbfb9c-insights-runtime-extractor-tls podName:2abb5699-2333-4c3f-b748-d3a227fbfb9c nodeName:}" failed. No retries permitted until 2026-07-27 21:04:25.699545368 +0000 UTC m=+57.831161574 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2abb5699-2333-4c3f-b748-d3a227fbfb9c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wkkhn" (UID: "2abb5699-2333-4c3f-b748-d3a227fbfb9c") : secret "insights-runtime-extractor-tls" not found Jul 27 21:04:23.742719 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:23.742666 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-j5rmn" Jul 27 21:04:23.742898 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:23.742790 2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-j5rmn" Jul 27 21:04:23.743179 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:23.743163 2577 scope.go:117] "RemoveContainer" containerID="010d6f34a18c34bbd4ce69002f8598033b36b045e6606777f010b27ef6844930" Jul 27 21:04:23.743351 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:23.743332 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-j5rmn_openshift-console-operator(4812c628-5f41-4e36-aaf6-8d62e4b6a443)\"" pod="openshift-console-operator/console-operator-584dcb6b54-j5rmn" podUID="4812c628-5f41-4e36-aaf6-8d62e4b6a443" Jul 27 21:04:24.514650 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:24.514622 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-npnjm_4c2ebad9-de50-4346-a64f-6cceecf44434/dns-node-resolver/0.log" Jul 27 21:04:25.514981 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:25.514951 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zt59k_e7bf9687-782f-40c9-bb73-e8b4b2c6a763/node-ca/0.log" Jul 27 21:04:25.721088 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:25.721048 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2abb5699-2333-4c3f-b748-d3a227fbfb9c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wkkhn\" (UID: \"2abb5699-2333-4c3f-b748-d3a227fbfb9c\") " pod="openshift-insights/insights-runtime-extractor-wkkhn" Jul 27 21:04:25.721248 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:25.721167 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 27 21:04:25.721248 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:25.721179 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2abb5699-2333-4c3f-b748-d3a227fbfb9c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wkkhn\" (UID: \"2abb5699-2333-4c3f-b748-d3a227fbfb9c\") " pod="openshift-insights/insights-runtime-extractor-wkkhn" Jul 27 21:04:25.721248 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:25.721227 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2abb5699-2333-4c3f-b748-d3a227fbfb9c-kube-rbac-proxy-cm podName:2abb5699-2333-4c3f-b748-d3a227fbfb9c nodeName:}" failed. No retries permitted until 2026-07-27 21:04:29.72121079 +0000 UTC m=+61.852826991 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2abb5699-2333-4c3f-b748-d3a227fbfb9c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wkkhn" (UID: "2abb5699-2333-4c3f-b748-d3a227fbfb9c") : configmap "kube-rbac-proxy" not found Jul 27 21:04:25.721369 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:25.721294 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 21:04:25.721369 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:25.721351 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2abb5699-2333-4c3f-b748-d3a227fbfb9c-insights-runtime-extractor-tls podName:2abb5699-2333-4c3f-b748-d3a227fbfb9c nodeName:}" failed. No retries permitted until 2026-07-27 21:04:29.721334672 +0000 UTC m=+61.852950888 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2abb5699-2333-4c3f-b748-d3a227fbfb9c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wkkhn" (UID: "2abb5699-2333-4c3f-b748-d3a227fbfb9c") : secret "insights-runtime-extractor-tls" not found Jul 27 21:04:25.915234 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:25.915158 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7gnng_597ea753-2547-4a21-9678-89da32128dcb/serve-healthcheck-canary/0.log" Jul 27 21:04:26.716668 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:26.716638 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-9rp9b_b2e7d196-079c-4bcd-aaf0-affbae8a0594/kube-storage-version-migrator-operator/0.log" Jul 27 21:04:29.051160 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:29.051119 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86e4b5a1-4521-43b3-9bf8-c42ec92d21f8-metrics-certs\") pod \"router-default-64c559777f-75n27\" (UID: \"86e4b5a1-4521-43b3-9bf8-c42ec92d21f8\") " pod="openshift-ingress/router-default-64c559777f-75n27" Jul 27 21:04:29.051612 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:29.051179 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86e4b5a1-4521-43b3-9bf8-c42ec92d21f8-service-ca-bundle\") pod \"router-default-64c559777f-75n27\" (UID: \"86e4b5a1-4521-43b3-9bf8-c42ec92d21f8\") " pod="openshift-ingress/router-default-64c559777f-75n27" Jul 27 21:04:29.051612 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:29.051273 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 27 21:04:29.051612 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:29.051319 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/86e4b5a1-4521-43b3-9bf8-c42ec92d21f8-service-ca-bundle podName:86e4b5a1-4521-43b3-9bf8-c42ec92d21f8 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:45.051298671 +0000 UTC m=+77.182914872 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/86e4b5a1-4521-43b3-9bf8-c42ec92d21f8-service-ca-bundle") pod "router-default-64c559777f-75n27" (UID: "86e4b5a1-4521-43b3-9bf8-c42ec92d21f8") : configmap references non-existent config key: service-ca.crt Jul 27 21:04:29.051612 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:29.051342 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86e4b5a1-4521-43b3-9bf8-c42ec92d21f8-metrics-certs podName:86e4b5a1-4521-43b3-9bf8-c42ec92d21f8 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:45.05133475 +0000 UTC m=+77.182950951 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86e4b5a1-4521-43b3-9bf8-c42ec92d21f8-metrics-certs") pod "router-default-64c559777f-75n27" (UID: "86e4b5a1-4521-43b3-9bf8-c42ec92d21f8") : secret "router-metrics-certs-default" not found Jul 27 21:04:29.152589 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:29.152525 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/91a29cde-048a-4e5e-a804-5854f86081ed-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-dm69z\" (UID: \"91a29cde-048a-4e5e-a804-5854f86081ed\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-dm69z" Jul 27 21:04:29.152767 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:29.152682 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 27 21:04:29.152767 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:29.152755 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91a29cde-048a-4e5e-a804-5854f86081ed-cluster-monitoring-operator-tls podName:91a29cde-048a-4e5e-a804-5854f86081ed nodeName:}" failed. No retries permitted until 2026-07-27 21:04:45.152739231 +0000 UTC m=+77.284355436 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/91a29cde-048a-4e5e-a804-5854f86081ed-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-dm69z" (UID: "91a29cde-048a-4e5e-a804-5854f86081ed") : secret "cluster-monitoring-operator-tls" not found Jul 27 21:04:29.253657 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:29.253620 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d1562fc4-9850-4084-b2d9-7b388a637f32-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-zwmht\" (UID: \"d1562fc4-9850-4084-b2d9-7b388a637f32\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-zwmht" Jul 27 21:04:29.256143 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:29.256113 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d1562fc4-9850-4084-b2d9-7b388a637f32-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-zwmht\" (UID: \"d1562fc4-9850-4084-b2d9-7b388a637f32\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-zwmht" Jul 27 21:04:29.318492 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:29.318401 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-fxz59\"" Jul 27 21:04:29.326575 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:29.326545 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-zwmht" Jul 27 21:04:29.354936 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:29.354895 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e066fd54-5796-438f-bdfc-784986a3713c-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-d2494\" (UID: \"e066fd54-5796-438f-bdfc-784986a3713c\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-d2494" Jul 27 21:04:29.355095 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:29.355039 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 27 21:04:29.355135 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:29.355107 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e066fd54-5796-438f-bdfc-784986a3713c-networking-console-plugin-cert podName:e066fd54-5796-438f-bdfc-784986a3713c nodeName:}" failed. No retries permitted until 2026-07-27 21:04:45.355091621 +0000 UTC m=+77.486707826 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e066fd54-5796-438f-bdfc-784986a3713c-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-d2494" (UID: "e066fd54-5796-438f-bdfc-784986a3713c") : secret "networking-console-plugin-cert" not found Jul 27 21:04:29.485675 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:29.485637 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-zwmht"] Jul 27 21:04:29.539048 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:29.539018 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l6mhn" Jul 27 21:04:29.659004 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:29.658922 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-zwmht" event={"ID":"d1562fc4-9850-4084-b2d9-7b388a637f32","Type":"ContainerStarted","Data":"91dbfe1eb461870d302362e1cc6e7467691fac1a5eaa001ca4864680a7c890e7"} Jul 27 21:04:29.758547 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:29.758507 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2abb5699-2333-4c3f-b748-d3a227fbfb9c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wkkhn\" (UID: \"2abb5699-2333-4c3f-b748-d3a227fbfb9c\") " pod="openshift-insights/insights-runtime-extractor-wkkhn" Jul 27 21:04:29.758743 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:29.758626 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2abb5699-2333-4c3f-b748-d3a227fbfb9c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wkkhn\" (UID: \"2abb5699-2333-4c3f-b748-d3a227fbfb9c\") " pod="openshift-insights/insights-runtime-extractor-wkkhn" Jul 27 21:04:29.758743 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:29.758656 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 27 21:04:29.758743 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:29.758733 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2abb5699-2333-4c3f-b748-d3a227fbfb9c-kube-rbac-proxy-cm podName:2abb5699-2333-4c3f-b748-d3a227fbfb9c nodeName:}" failed. No retries permitted until 2026-07-27 21:04:37.758711725 +0000 UTC m=+69.890327926 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2abb5699-2333-4c3f-b748-d3a227fbfb9c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wkkhn" (UID: "2abb5699-2333-4c3f-b748-d3a227fbfb9c") : configmap "kube-rbac-proxy" not found Jul 27 21:04:29.758743 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:29.758743 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 21:04:29.758912 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:29.758802 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2abb5699-2333-4c3f-b748-d3a227fbfb9c-insights-runtime-extractor-tls podName:2abb5699-2333-4c3f-b748-d3a227fbfb9c nodeName:}" failed. No retries permitted until 2026-07-27 21:04:37.758790388 +0000 UTC m=+69.890406589 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2abb5699-2333-4c3f-b748-d3a227fbfb9c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wkkhn" (UID: "2abb5699-2333-4c3f-b748-d3a227fbfb9c") : secret "insights-runtime-extractor-tls" not found Jul 27 21:04:31.667976 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:31.667870 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-zwmht" event={"ID":"d1562fc4-9850-4084-b2d9-7b388a637f32","Type":"ContainerStarted","Data":"085e3b41ed220a60b5808ade42cf3b1b41b77e55c1dea9c162c41decd8764a31"} Jul 27 21:04:31.667976 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:31.667941 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-zwmht" event={"ID":"d1562fc4-9850-4084-b2d9-7b388a637f32","Type":"ContainerStarted","Data":"ae1630347c894a8fc84dec433eca4caf9e452244f11f2815cd0d39c7b161eda0"} Jul 27 21:04:31.688516 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:31.688455 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-zwmht" podStartSLOduration=16.92883575 podStartE2EDuration="18.688438088s" totalImagesPullingTime="1.759602338s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:04:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:31.687114485 +0000 UTC m=+63.818730710" watchObservedRunningTime="2026-07-27 21:04:31.688438088 +0000 UTC m=+63.820054366" Jul 27 21:04:34.099409 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:34.099341 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c77555ba-462e-49c5-b8b0-e83987e8a2e3-original-pull-secret\") pod \"global-pull-secret-syncer-l2czx\" (UID: \"c77555ba-462e-49c5-b8b0-e83987e8a2e3\") " pod="kube-system/global-pull-secret-syncer-l2czx" Jul 27 21:04:34.099867 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:34.099481 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c32154a-b41d-45c4-afa6-240a8ea24835-metrics-certs\") pod \"network-metrics-daemon-lvg8z\" (UID: \"0c32154a-b41d-45c4-afa6-240a8ea24835\") " pod="openshift-multus/network-metrics-daemon-lvg8z" Jul 27 21:04:34.102075 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:34.102043 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 27 21:04:34.102213 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:34.102048 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 27 21:04:34.109941 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:34.109912 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 27 21:04:34.110052 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:34.110040 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c32154a-b41d-45c4-afa6-240a8ea24835-metrics-certs podName:0c32154a-b41d-45c4-afa6-240a8ea24835 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:38.110022331 +0000 UTC m=+130.241638536 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0c32154a-b41d-45c4-afa6-240a8ea24835-metrics-certs") pod "network-metrics-daemon-lvg8z" (UID: "0c32154a-b41d-45c4-afa6-240a8ea24835") : secret "metrics-daemon-secret" not found Jul 27 21:04:34.112166 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:34.112145 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c77555ba-462e-49c5-b8b0-e83987e8a2e3-original-pull-secret\") pod \"global-pull-secret-syncer-l2czx\" (UID: \"c77555ba-462e-49c5-b8b0-e83987e8a2e3\") " pod="kube-system/global-pull-secret-syncer-l2czx" Jul 27 21:04:34.200664 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:34.200625 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft79h\" (UniqueName: \"kubernetes.io/projected/8157bc07-d223-44de-aee6-ffa0c683a2cb-kube-api-access-ft79h\") pod \"network-check-target-lrnv5\" (UID: \"8157bc07-d223-44de-aee6-ffa0c683a2cb\") " pod="openshift-network-diagnostics/network-check-target-lrnv5" Jul 27 21:04:34.203185 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:34.203157 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft79h\" (UniqueName: \"kubernetes.io/projected/8157bc07-d223-44de-aee6-ffa0c683a2cb-kube-api-access-ft79h\") pod \"network-check-target-lrnv5\" (UID: \"8157bc07-d223-44de-aee6-ffa0c683a2cb\") " pod="openshift-network-diagnostics/network-check-target-lrnv5" Jul 27 21:04:34.373992 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:34.373906 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ww7jp\"" Jul 27 21:04:34.377870 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:34.377850 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l2czx" Jul 27 21:04:34.382021 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:34.381989 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lrnv5" Jul 27 21:04:34.539434 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:34.539407 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-l2czx"] Jul 27 21:04:34.541833 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:04:34.541806 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc77555ba_462e_49c5_b8b0_e83987e8a2e3.slice/crio-7746e4af00282223494aabe3654399b1e1af2f9978e4ab77a1040dba2051c450 WatchSource:0}: Error finding container 7746e4af00282223494aabe3654399b1e1af2f9978e4ab77a1040dba2051c450: Status 404 returned error can't find the container with id 7746e4af00282223494aabe3654399b1e1af2f9978e4ab77a1040dba2051c450 Jul 27 21:04:34.556683 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:34.556655 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lrnv5"] Jul 27 21:04:34.560684 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:04:34.560644 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8157bc07_d223_44de_aee6_ffa0c683a2cb.slice/crio-ff8f8d4bf9fc07b63d462b7f022a798b76a2e16f8290edcc750c7a2a805b2ba2 WatchSource:0}: Error finding container ff8f8d4bf9fc07b63d462b7f022a798b76a2e16f8290edcc750c7a2a805b2ba2: Status 404 returned error can't find the container with id ff8f8d4bf9fc07b63d462b7f022a798b76a2e16f8290edcc750c7a2a805b2ba2 Jul 27 21:04:34.678020 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:34.677921 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lrnv5" event={"ID":"8157bc07-d223-44de-aee6-ffa0c683a2cb","Type":"ContainerStarted","Data":"cf1000969ebf5a7156c4692a3a5677e5d4c9b14017f7c73eb2a4fc14713de9a8"} Jul 27 21:04:34.678020 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:34.677973 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lrnv5" event={"ID":"8157bc07-d223-44de-aee6-ffa0c683a2cb","Type":"ContainerStarted","Data":"ff8f8d4bf9fc07b63d462b7f022a798b76a2e16f8290edcc750c7a2a805b2ba2"} Jul 27 21:04:34.678255 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:34.678022 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-lrnv5" Jul 27 21:04:34.679169 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:34.679145 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-l2czx" event={"ID":"c77555ba-462e-49c5-b8b0-e83987e8a2e3","Type":"ContainerStarted","Data":"7746e4af00282223494aabe3654399b1e1af2f9978e4ab77a1040dba2051c450"} Jul 27 21:04:34.697563 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:34.697501 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-lrnv5" podStartSLOduration=65.697479088 podStartE2EDuration="1m5.697479088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:03:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:34.696437295 +0000 UTC m=+66.828053517" watchObservedRunningTime="2026-07-27 21:04:34.697479088 +0000 UTC m=+66.829095368" Jul 27 21:04:35.361096 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:35.361055 2577 scope.go:117] "RemoveContainer" containerID="010d6f34a18c34bbd4ce69002f8598033b36b045e6606777f010b27ef6844930" Jul 27 21:04:35.685335 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:35.685247 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-j5rmn_4812c628-5f41-4e36-aaf6-8d62e4b6a443/console-operator/2.log" Jul 27 21:04:35.685788 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:35.685766 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-j5rmn_4812c628-5f41-4e36-aaf6-8d62e4b6a443/console-operator/1.log" Jul 27 21:04:35.685887 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:35.685810 2577 generic.go:356] "Generic (PLEG): container finished" podID="4812c628-5f41-4e36-aaf6-8d62e4b6a443" containerID="4405d2e6c65d430a7fc45b6d010573fdb2b668d3a1670c71a5f26c8d00695a6c" exitCode=255 Jul 27 21:04:35.685937 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:35.685887 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-j5rmn" event={"ID":"4812c628-5f41-4e36-aaf6-8d62e4b6a443","Type":"ContainerDied","Data":"4405d2e6c65d430a7fc45b6d010573fdb2b668d3a1670c71a5f26c8d00695a6c"} Jul 27 21:04:35.685986 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:35.685959 2577 scope.go:117] "RemoveContainer" containerID="010d6f34a18c34bbd4ce69002f8598033b36b045e6606777f010b27ef6844930" Jul 27 21:04:35.686491 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:35.686467 2577 scope.go:117] "RemoveContainer" containerID="4405d2e6c65d430a7fc45b6d010573fdb2b668d3a1670c71a5f26c8d00695a6c" Jul 27 21:04:35.686713 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:35.686685 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-584dcb6b54-j5rmn_openshift-console-operator(4812c628-5f41-4e36-aaf6-8d62e4b6a443)\"" pod="openshift-console-operator/console-operator-584dcb6b54-j5rmn" podUID="4812c628-5f41-4e36-aaf6-8d62e4b6a443" Jul 27 21:04:36.219028 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:36.218990 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/932ad643-a059-44a4-90f8-41016fc1e5c0-metrics-tls\") pod \"dns-default-ppf68\" (UID: \"932ad643-a059-44a4-90f8-41016fc1e5c0\") " pod="openshift-dns/dns-default-ppf68" Jul 27 21:04:36.219273 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:36.219051 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eb9ca170-7ea1-45de-aca4-54b4731ed82d-registry-tls\") pod \"image-registry-75fb88d74f-jhdnw\" (UID: \"eb9ca170-7ea1-45de-aca4-54b4731ed82d\") " pod="openshift-image-registry/image-registry-75fb88d74f-jhdnw" Jul 27 21:04:36.221547 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:36.221520 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/932ad643-a059-44a4-90f8-41016fc1e5c0-metrics-tls\") pod \"dns-default-ppf68\" (UID: \"932ad643-a059-44a4-90f8-41016fc1e5c0\") " pod="openshift-dns/dns-default-ppf68" Jul 27 21:04:36.221687 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:36.221586 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eb9ca170-7ea1-45de-aca4-54b4731ed82d-registry-tls\") pod \"image-registry-75fb88d74f-jhdnw\" (UID: \"eb9ca170-7ea1-45de-aca4-54b4731ed82d\") " pod="openshift-image-registry/image-registry-75fb88d74f-jhdnw" Jul 27 21:04:36.378266 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:36.378229 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-l946f\"" Jul 27 21:04:36.386583 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:36.386545 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75fb88d74f-jhdnw" Jul 27 21:04:36.388009 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:36.387978 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jkr4s\"" Jul 27 21:04:36.396153 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:36.396117 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ppf68" Jul 27 21:04:36.580697 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:36.580664 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-75fb88d74f-jhdnw"] Jul 27 21:04:36.583779 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:04:36.583733 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb9ca170_7ea1_45de_aca4_54b4731ed82d.slice/crio-69b1c381a6c137daa757306a9262bcca7b1b5f9d8cc474676398dd5f8b2e7b49 WatchSource:0}: Error finding container 69b1c381a6c137daa757306a9262bcca7b1b5f9d8cc474676398dd5f8b2e7b49: Status 404 returned error can't find the container with id 69b1c381a6c137daa757306a9262bcca7b1b5f9d8cc474676398dd5f8b2e7b49 Jul 27 21:04:36.614477 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:36.614443 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ppf68"] Jul 27 21:04:36.618657 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:04:36.618623 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod932ad643_a059_44a4_90f8_41016fc1e5c0.slice/crio-fcf76afc7eb7d27784937648c20d27af91bceefc6d7246f8fe5a5d4be62bc468 WatchSource:0}: Error finding container fcf76afc7eb7d27784937648c20d27af91bceefc6d7246f8fe5a5d4be62bc468: Status 404 returned error can't find the container with id fcf76afc7eb7d27784937648c20d27af91bceefc6d7246f8fe5a5d4be62bc468 Jul 27 21:04:36.693617 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:36.693563 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-j5rmn_4812c628-5f41-4e36-aaf6-8d62e4b6a443/console-operator/2.log" Jul 27 21:04:36.694895 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:36.694857 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ppf68" event={"ID":"932ad643-a059-44a4-90f8-41016fc1e5c0","Type":"ContainerStarted","Data":"fcf76afc7eb7d27784937648c20d27af91bceefc6d7246f8fe5a5d4be62bc468"} Jul 27 21:04:36.696112 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:36.696069 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75fb88d74f-jhdnw" event={"ID":"eb9ca170-7ea1-45de-aca4-54b4731ed82d","Type":"ContainerStarted","Data":"69b1c381a6c137daa757306a9262bcca7b1b5f9d8cc474676398dd5f8b2e7b49"} Jul 27 21:04:37.706919 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:37.706873 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75fb88d74f-jhdnw" event={"ID":"eb9ca170-7ea1-45de-aca4-54b4731ed82d","Type":"ContainerStarted","Data":"ac2ec9993d1aab75ccd7a17254aea313a695d570933ebc61805e6d5f1d135581"} Jul 27 21:04:37.707381 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:37.706999 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-75fb88d74f-jhdnw" Jul 27 21:04:37.735095 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:37.735029 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-75fb88d74f-jhdnw" podStartSLOduration=68.735008618 podStartE2EDuration="1m8.735008618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:03:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:37.733729549 +0000 UTC m=+69.865345773" watchObservedRunningTime="2026-07-27 21:04:37.735008618 +0000 UTC m=+69.866624840" Jul 27 21:04:37.834494 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:37.834438 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2abb5699-2333-4c3f-b748-d3a227fbfb9c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wkkhn\" (UID: \"2abb5699-2333-4c3f-b748-d3a227fbfb9c\") " pod="openshift-insights/insights-runtime-extractor-wkkhn" Jul 27 21:04:37.835031 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:37.834994 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2abb5699-2333-4c3f-b748-d3a227fbfb9c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wkkhn\" (UID: \"2abb5699-2333-4c3f-b748-d3a227fbfb9c\") " pod="openshift-insights/insights-runtime-extractor-wkkhn" Jul 27 21:04:37.835184 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:37.835079 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2abb5699-2333-4c3f-b748-d3a227fbfb9c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wkkhn\" (UID: \"2abb5699-2333-4c3f-b748-d3a227fbfb9c\") " pod="openshift-insights/insights-runtime-extractor-wkkhn" Jul 27 21:04:37.837778 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:37.837748 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2abb5699-2333-4c3f-b748-d3a227fbfb9c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wkkhn\" (UID: \"2abb5699-2333-4c3f-b748-d3a227fbfb9c\") " pod="openshift-insights/insights-runtime-extractor-wkkhn" Jul 27 21:04:37.875258 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:37.875213 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-w5dc2\"" Jul 27 21:04:37.882277 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:37.882242 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wkkhn" Jul 27 21:04:39.355440 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:39.355397 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wkkhn"] Jul 27 21:04:39.360779 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:04:39.360740 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2abb5699_2333_4c3f_b748_d3a227fbfb9c.slice/crio-d1ca1c953fce824a602f5b54a51d54535b10623ffeacf4e66de919554eae6360 WatchSource:0}: Error finding container d1ca1c953fce824a602f5b54a51d54535b10623ffeacf4e66de919554eae6360: Status 404 returned error can't find the container with id d1ca1c953fce824a602f5b54a51d54535b10623ffeacf4e66de919554eae6360 Jul 27 21:04:39.715891 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:39.715848 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wkkhn" event={"ID":"2abb5699-2333-4c3f-b748-d3a227fbfb9c","Type":"ContainerStarted","Data":"d1ca1c953fce824a602f5b54a51d54535b10623ffeacf4e66de919554eae6360"} Jul 27 21:04:39.717454 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:39.717423 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-l2czx" event={"ID":"c77555ba-462e-49c5-b8b0-e83987e8a2e3","Type":"ContainerStarted","Data":"f84fa3416c81e7582f5dfd14183d33e703efd5e0891fc163146b09fe9b3ebcaa"} Jul 27 21:04:39.719082 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:39.719055 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ppf68" event={"ID":"932ad643-a059-44a4-90f8-41016fc1e5c0","Type":"ContainerStarted","Data":"2e465b838e3606819b619cf9e2d687d01a21adf4ed93dda9970d12bfa8ee9a69"} Jul 27 21:04:39.719201 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:39.719099 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ppf68" event={"ID":"932ad643-a059-44a4-90f8-41016fc1e5c0","Type":"ContainerStarted","Data":"deb5518219b00d6852d016fc661d95ff58e4591ace7ca78b72a77f6ba3174f03"} Jul 27 21:04:39.719201 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:39.719176 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-ppf68" Jul 27 21:04:39.734094 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:39.734040 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-l2czx" podStartSLOduration=66.054664891 podStartE2EDuration="1m10.734020477s" totalImagesPullingTime="4.679355586s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:03:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:39.733071426 +0000 UTC m=+71.864687649" watchObservedRunningTime="2026-07-27 21:04:39.734020477 +0000 UTC m=+71.865636683" Jul 27 21:04:39.750926 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:39.750866 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-ppf68" podStartSLOduration=33.154430492 podStartE2EDuration="35.750844639s" totalImagesPullingTime="2.596414147s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:04:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:39.750605494 +0000 UTC m=+71.882221713" watchObservedRunningTime="2026-07-27 21:04:39.750844639 +0000 UTC m=+71.882460861" Jul 27 21:04:43.742653 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:43.742614 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-j5rmn" Jul 27 21:04:43.743130 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:43.742684 2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-j5rmn" Jul 27 21:04:43.743130 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:43.743044 2577 scope.go:117] "RemoveContainer" containerID="4405d2e6c65d430a7fc45b6d010573fdb2b668d3a1670c71a5f26c8d00695a6c" Jul 27 21:04:43.747181 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:43.747148 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-584dcb6b54-j5rmn_openshift-console-operator(4812c628-5f41-4e36-aaf6-8d62e4b6a443)\"" pod="openshift-console-operator/console-operator-584dcb6b54-j5rmn" podUID="4812c628-5f41-4e36-aaf6-8d62e4b6a443" Jul 27 21:04:44.743489 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:44.743449 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wkkhn" event={"ID":"2abb5699-2333-4c3f-b748-d3a227fbfb9c","Type":"ContainerStarted","Data":"8df910e8d92945d9552ce54fea54b5ccceb993bc7b8ca0e82fe479c71e935d61"} Jul 27 21:04:45.105598 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:45.105465 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86e4b5a1-4521-43b3-9bf8-c42ec92d21f8-metrics-certs\") pod \"router-default-64c559777f-75n27\" (UID: \"86e4b5a1-4521-43b3-9bf8-c42ec92d21f8\") " pod="openshift-ingress/router-default-64c559777f-75n27" Jul 27 21:04:45.105598 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:45.105550 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86e4b5a1-4521-43b3-9bf8-c42ec92d21f8-service-ca-bundle\") pod \"router-default-64c559777f-75n27\" (UID: \"86e4b5a1-4521-43b3-9bf8-c42ec92d21f8\") " pod="openshift-ingress/router-default-64c559777f-75n27" Jul 27 21:04:45.106421 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:45.106366 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86e4b5a1-4521-43b3-9bf8-c42ec92d21f8-service-ca-bundle\") pod \"router-default-64c559777f-75n27\" (UID: \"86e4b5a1-4521-43b3-9bf8-c42ec92d21f8\") " pod="openshift-ingress/router-default-64c559777f-75n27" Jul 27 21:04:45.108893 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:45.108839 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86e4b5a1-4521-43b3-9bf8-c42ec92d21f8-metrics-certs\") pod \"router-default-64c559777f-75n27\" (UID: \"86e4b5a1-4521-43b3-9bf8-c42ec92d21f8\") " pod="openshift-ingress/router-default-64c559777f-75n27" Jul 27 21:04:45.206874 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:45.206801 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/91a29cde-048a-4e5e-a804-5854f86081ed-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-dm69z\" (UID: \"91a29cde-048a-4e5e-a804-5854f86081ed\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-dm69z" Jul 27 21:04:45.209970 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:45.209895 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/91a29cde-048a-4e5e-a804-5854f86081ed-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-dm69z\" (UID: \"91a29cde-048a-4e5e-a804-5854f86081ed\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-dm69z" Jul 27 21:04:45.281128 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:45.281099 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-4zpl8\"" Jul 27 21:04:45.290059 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:45.290026 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-64c559777f-75n27" Jul 27 21:04:45.389274 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:45.389248 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-5r8h2\"" Jul 27 21:04:45.397447 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:45.397409 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-dm69z" Jul 27 21:04:45.408760 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:45.408718 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e066fd54-5796-438f-bdfc-784986a3713c-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-d2494\" (UID: \"e066fd54-5796-438f-bdfc-784986a3713c\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-d2494" Jul 27 21:04:45.412523 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:45.412452 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e066fd54-5796-438f-bdfc-784986a3713c-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-d2494\" (UID: \"e066fd54-5796-438f-bdfc-784986a3713c\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-d2494" Jul 27 21:04:45.450340 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:45.450298 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-64c559777f-75n27"] Jul 27 21:04:45.454393 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:04:45.454359 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86e4b5a1_4521_43b3_9bf8_c42ec92d21f8.slice/crio-a1a2b7bc640f6ed0a8525cf897649a5af36422accc5f000029fadb6a3f46acd3 WatchSource:0}: Error finding container a1a2b7bc640f6ed0a8525cf897649a5af36422accc5f000029fadb6a3f46acd3: Status 404 returned error can't find the container with id a1a2b7bc640f6ed0a8525cf897649a5af36422accc5f000029fadb6a3f46acd3 Jul 27 21:04:45.525437 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:45.525408 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-bj9tf\"" Jul 27 21:04:45.533702 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:45.533661 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-d2494" Jul 27 21:04:45.563758 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:45.563723 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-dm69z"] Jul 27 21:04:45.568933 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:04:45.568901 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91a29cde_048a_4e5e_a804_5854f86081ed.slice/crio-90921512a69f924d57dc1904b947b7e18d61a846c4d476b74b700af0afc4dd93 WatchSource:0}: Error finding container 90921512a69f924d57dc1904b947b7e18d61a846c4d476b74b700af0afc4dd93: Status 404 returned error can't find the container with id 90921512a69f924d57dc1904b947b7e18d61a846c4d476b74b700af0afc4dd93 Jul 27 21:04:45.694490 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:45.694430 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-d2494"] Jul 27 21:04:45.697657 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:04:45.697615 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode066fd54_5796_438f_bdfc_784986a3713c.slice/crio-147fe053aede9682cd4cf2c8eb3b953e77c31890b05cfa2903622e2c9549e39f WatchSource:0}: Error finding container 147fe053aede9682cd4cf2c8eb3b953e77c31890b05cfa2903622e2c9549e39f: Status 404 returned error can't find the container with id 147fe053aede9682cd4cf2c8eb3b953e77c31890b05cfa2903622e2c9549e39f Jul 27 21:04:45.749070 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:45.749031 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-d2494" event={"ID":"e066fd54-5796-438f-bdfc-784986a3713c","Type":"ContainerStarted","Data":"147fe053aede9682cd4cf2c8eb3b953e77c31890b05cfa2903622e2c9549e39f"} Jul 27 21:04:45.750319 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:45.750292 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-dm69z" event={"ID":"91a29cde-048a-4e5e-a804-5854f86081ed","Type":"ContainerStarted","Data":"90921512a69f924d57dc1904b947b7e18d61a846c4d476b74b700af0afc4dd93"} Jul 27 21:04:45.751960 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:45.751929 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-64c559777f-75n27" event={"ID":"86e4b5a1-4521-43b3-9bf8-c42ec92d21f8","Type":"ContainerStarted","Data":"1f91126fd9621c534994b649b0c653c1464608320d582c7595980393bae457a5"} Jul 27 21:04:45.752066 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:45.751975 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-64c559777f-75n27" event={"ID":"86e4b5a1-4521-43b3-9bf8-c42ec92d21f8","Type":"ContainerStarted","Data":"a1a2b7bc640f6ed0a8525cf897649a5af36422accc5f000029fadb6a3f46acd3"} Jul 27 21:04:45.773070 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:45.773010 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-64c559777f-75n27" podStartSLOduration=32.772990973 podStartE2EDuration="32.772990973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:04:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:45.771278337 +0000 UTC m=+77.902894561" watchObservedRunningTime="2026-07-27 21:04:45.772990973 +0000 UTC m=+77.904607197" Jul 27 21:04:46.290397 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:46.290353 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-64c559777f-75n27" Jul 27 21:04:46.293589 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:46.293544 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-64c559777f-75n27" Jul 27 21:04:46.756760 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:46.756720 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-64c559777f-75n27" Jul 27 21:04:46.758222 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:46.758191 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-64c559777f-75n27" Jul 27 21:04:49.726351 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:49.726322 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-ppf68" Jul 27 21:04:49.771653 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:49.771614 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-d2494" event={"ID":"e066fd54-5796-438f-bdfc-784986a3713c","Type":"ContainerStarted","Data":"016a3fe70ef614b205030c79af4546cb508d3d5d115dc5af7183f29b14e32605"} Jul 27 21:04:49.773712 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:49.773680 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-dm69z" event={"ID":"91a29cde-048a-4e5e-a804-5854f86081ed","Type":"ContainerStarted","Data":"3aedf906d7722beb5b20713d2fbc3ebe4a778c906b3b2c9d92f568a5a745271d"} Jul 27 21:04:49.776064 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:49.775984 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wkkhn" event={"ID":"2abb5699-2333-4c3f-b748-d3a227fbfb9c","Type":"ContainerStarted","Data":"157c1ebaa742e672f8b238b36cdc7fa02d30c147297eb4b92ef45857b873a3e0"} Jul 27 21:04:49.797187 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:49.797028 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-d2494" podStartSLOduration=33.649729881 podStartE2EDuration="36.797004564s" totalImagesPullingTime="3.147274683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:04:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:49.794475599 +0000 UTC m=+81.926091824" watchObservedRunningTime="2026-07-27 21:04:49.797004564 +0000 UTC m=+81.928620788" Jul 27 21:04:49.817451 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:49.817387 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-dm69z" podStartSLOduration=33.544297927 podStartE2EDuration="36.817365251s" totalImagesPullingTime="3.273067324s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:04:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:49.815865868 +0000 UTC m=+81.947482091" watchObservedRunningTime="2026-07-27 21:04:49.817365251 +0000 UTC m=+81.948981475" Jul 27 21:04:53.793817 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:53.793780 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wkkhn" event={"ID":"2abb5699-2333-4c3f-b748-d3a227fbfb9c","Type":"ContainerStarted","Data":"9d42fd1492724cc92267aca2334b47a11ff47f6350c258b5b3e932c5b65ef461"} Jul 27 21:04:53.812180 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:53.812127 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-wkkhn" podStartSLOduration=19.395327613 podStartE2EDuration="32.81210775s" totalImagesPullingTime="13.416780137s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:04:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:53.810812154 +0000 UTC m=+85.942428378" watchObservedRunningTime="2026-07-27 21:04:53.81210775 +0000 UTC m=+85.943723972" Jul 27 21:04:55.361497 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:55.361463 2577 scope.go:117] "RemoveContainer" containerID="4405d2e6c65d430a7fc45b6d010573fdb2b668d3a1670c71a5f26c8d00695a6c" Jul 27 21:04:55.361939 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:04:55.361669 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-584dcb6b54-j5rmn_openshift-console-operator(4812c628-5f41-4e36-aaf6-8d62e4b6a443)\"" pod="openshift-console-operator/console-operator-584dcb6b54-j5rmn" podUID="4812c628-5f41-4e36-aaf6-8d62e4b6a443" Jul 27 21:04:58.714622 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:04:58.714555 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-75fb88d74f-jhdnw" Jul 27 21:05:01.845943 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:01.845902 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-q9q9v"] Jul 27 21:05:01.849775 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:01.849752 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-q9q9v" Jul 27 21:05:01.852087 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:01.852064 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 27 21:05:01.852217 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:01.852106 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 27 21:05:01.852906 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:01.852890 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 27 21:05:01.853126 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:01.853113 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 27 21:05:01.853857 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:01.853832 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ntxzb\"" Jul 27 21:05:01.951683 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:01.951642 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f0119fc2-7a0d-4f10-ae6a-e583297b08e4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q9q9v\" (UID: \"f0119fc2-7a0d-4f10-ae6a-e583297b08e4\") " pod="openshift-monitoring/node-exporter-q9q9v" Jul 27 21:05:01.951683 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:01.951691 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f0119fc2-7a0d-4f10-ae6a-e583297b08e4-root\") pod \"node-exporter-q9q9v\" (UID: \"f0119fc2-7a0d-4f10-ae6a-e583297b08e4\") " pod="openshift-monitoring/node-exporter-q9q9v" Jul 27 21:05:01.951939 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:01.951757 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f0119fc2-7a0d-4f10-ae6a-e583297b08e4-sys\") pod \"node-exporter-q9q9v\" (UID: \"f0119fc2-7a0d-4f10-ae6a-e583297b08e4\") " pod="openshift-monitoring/node-exporter-q9q9v" Jul 27 21:05:01.951939 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:01.951793 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4xg5\" (UniqueName: \"kubernetes.io/projected/f0119fc2-7a0d-4f10-ae6a-e583297b08e4-kube-api-access-x4xg5\") pod \"node-exporter-q9q9v\" (UID: \"f0119fc2-7a0d-4f10-ae6a-e583297b08e4\") " pod="openshift-monitoring/node-exporter-q9q9v" Jul 27 21:05:01.951939 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:01.951829 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f0119fc2-7a0d-4f10-ae6a-e583297b08e4-metrics-client-ca\") pod \"node-exporter-q9q9v\" (UID: \"f0119fc2-7a0d-4f10-ae6a-e583297b08e4\") " pod="openshift-monitoring/node-exporter-q9q9v" Jul 27 21:05:01.951939 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:01.951851 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f0119fc2-7a0d-4f10-ae6a-e583297b08e4-node-exporter-textfile\") pod \"node-exporter-q9q9v\" (UID: \"f0119fc2-7a0d-4f10-ae6a-e583297b08e4\") " pod="openshift-monitoring/node-exporter-q9q9v" Jul 27 21:05:01.951939 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:01.951871 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f0119fc2-7a0d-4f10-ae6a-e583297b08e4-node-exporter-wtmp\") pod \"node-exporter-q9q9v\" (UID: \"f0119fc2-7a0d-4f10-ae6a-e583297b08e4\") " pod="openshift-monitoring/node-exporter-q9q9v" Jul 27 21:05:01.952121 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:01.951951 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f0119fc2-7a0d-4f10-ae6a-e583297b08e4-node-exporter-tls\") pod \"node-exporter-q9q9v\" (UID: \"f0119fc2-7a0d-4f10-ae6a-e583297b08e4\") " pod="openshift-monitoring/node-exporter-q9q9v" Jul 27 21:05:01.952121 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:01.951995 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f0119fc2-7a0d-4f10-ae6a-e583297b08e4-node-exporter-accelerators-collector-config\") pod \"node-exporter-q9q9v\" (UID: \"f0119fc2-7a0d-4f10-ae6a-e583297b08e4\") " pod="openshift-monitoring/node-exporter-q9q9v" Jul 27 21:05:02.052761 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:02.052725 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f0119fc2-7a0d-4f10-ae6a-e583297b08e4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q9q9v\" (UID: \"f0119fc2-7a0d-4f10-ae6a-e583297b08e4\") " pod="openshift-monitoring/node-exporter-q9q9v" Jul 27 21:05:02.052961 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:02.052772 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f0119fc2-7a0d-4f10-ae6a-e583297b08e4-root\") pod \"node-exporter-q9q9v\" (UID: \"f0119fc2-7a0d-4f10-ae6a-e583297b08e4\") " pod="openshift-monitoring/node-exporter-q9q9v" Jul 27 21:05:02.052961 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:02.052806 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f0119fc2-7a0d-4f10-ae6a-e583297b08e4-sys\") pod \"node-exporter-q9q9v\" (UID: \"f0119fc2-7a0d-4f10-ae6a-e583297b08e4\") " pod="openshift-monitoring/node-exporter-q9q9v" Jul 27 21:05:02.052961 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:02.052826 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4xg5\" (UniqueName: \"kubernetes.io/projected/f0119fc2-7a0d-4f10-ae6a-e583297b08e4-kube-api-access-x4xg5\") pod \"node-exporter-q9q9v\" (UID: \"f0119fc2-7a0d-4f10-ae6a-e583297b08e4\") " pod="openshift-monitoring/node-exporter-q9q9v" Jul 27 21:05:02.052961 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:02.052866 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f0119fc2-7a0d-4f10-ae6a-e583297b08e4-metrics-client-ca\") pod \"node-exporter-q9q9v\" (UID: \"f0119fc2-7a0d-4f10-ae6a-e583297b08e4\") " pod="openshift-monitoring/node-exporter-q9q9v" Jul 27 21:05:02.052961 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:02.052890 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f0119fc2-7a0d-4f10-ae6a-e583297b08e4-node-exporter-textfile\") pod \"node-exporter-q9q9v\" (UID: \"f0119fc2-7a0d-4f10-ae6a-e583297b08e4\") " pod="openshift-monitoring/node-exporter-q9q9v" Jul 27 21:05:02.052961 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:02.052913 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f0119fc2-7a0d-4f10-ae6a-e583297b08e4-node-exporter-wtmp\") pod \"node-exporter-q9q9v\" (UID: \"f0119fc2-7a0d-4f10-ae6a-e583297b08e4\") " pod="openshift-monitoring/node-exporter-q9q9v" Jul 27 21:05:02.052961 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:02.052915 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f0119fc2-7a0d-4f10-ae6a-e583297b08e4-root\") pod \"node-exporter-q9q9v\" (UID: \"f0119fc2-7a0d-4f10-ae6a-e583297b08e4\") " pod="openshift-monitoring/node-exporter-q9q9v" Jul 27 21:05:02.052961 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:02.052929 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f0119fc2-7a0d-4f10-ae6a-e583297b08e4-sys\") pod \"node-exporter-q9q9v\" (UID: \"f0119fc2-7a0d-4f10-ae6a-e583297b08e4\") " pod="openshift-monitoring/node-exporter-q9q9v" Jul 27 21:05:02.052961 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:02.052948 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f0119fc2-7a0d-4f10-ae6a-e583297b08e4-node-exporter-tls\") pod \"node-exporter-q9q9v\" (UID: \"f0119fc2-7a0d-4f10-ae6a-e583297b08e4\") " pod="openshift-monitoring/node-exporter-q9q9v" Jul 27 21:05:02.053316 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:02.052970 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f0119fc2-7a0d-4f10-ae6a-e583297b08e4-node-exporter-accelerators-collector-config\") pod \"node-exporter-q9q9v\" (UID: \"f0119fc2-7a0d-4f10-ae6a-e583297b08e4\") " pod="openshift-monitoring/node-exporter-q9q9v" Jul 27 21:05:02.053316 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:02.053072 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f0119fc2-7a0d-4f10-ae6a-e583297b08e4-node-exporter-wtmp\") pod \"node-exporter-q9q9v\" (UID: \"f0119fc2-7a0d-4f10-ae6a-e583297b08e4\") " pod="openshift-monitoring/node-exporter-q9q9v" Jul 27 21:05:02.053533 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:02.053505 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f0119fc2-7a0d-4f10-ae6a-e583297b08e4-node-exporter-textfile\") pod \"node-exporter-q9q9v\" (UID: \"f0119fc2-7a0d-4f10-ae6a-e583297b08e4\") " pod="openshift-monitoring/node-exporter-q9q9v" Jul 27 21:05:02.053755 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:02.053733 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f0119fc2-7a0d-4f10-ae6a-e583297b08e4-metrics-client-ca\") pod \"node-exporter-q9q9v\" (UID: \"f0119fc2-7a0d-4f10-ae6a-e583297b08e4\") " pod="openshift-monitoring/node-exporter-q9q9v" Jul 27 21:05:02.053854 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:02.053832 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f0119fc2-7a0d-4f10-ae6a-e583297b08e4-node-exporter-accelerators-collector-config\") pod \"node-exporter-q9q9v\" (UID: \"f0119fc2-7a0d-4f10-ae6a-e583297b08e4\") " pod="openshift-monitoring/node-exporter-q9q9v" Jul 27 21:05:02.055313 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:02.055287 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f0119fc2-7a0d-4f10-ae6a-e583297b08e4-node-exporter-tls\") pod \"node-exporter-q9q9v\" (UID: \"f0119fc2-7a0d-4f10-ae6a-e583297b08e4\") " pod="openshift-monitoring/node-exporter-q9q9v" Jul 27 21:05:02.055395 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:02.055375 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f0119fc2-7a0d-4f10-ae6a-e583297b08e4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q9q9v\" (UID: \"f0119fc2-7a0d-4f10-ae6a-e583297b08e4\") " pod="openshift-monitoring/node-exporter-q9q9v" Jul 27 21:05:02.064545 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:02.064517 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4xg5\" (UniqueName: \"kubernetes.io/projected/f0119fc2-7a0d-4f10-ae6a-e583297b08e4-kube-api-access-x4xg5\") pod \"node-exporter-q9q9v\" (UID: \"f0119fc2-7a0d-4f10-ae6a-e583297b08e4\") " pod="openshift-monitoring/node-exporter-q9q9v" Jul 27 21:05:02.159327 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:02.159231 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-q9q9v" Jul 27 21:05:02.170682 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:05:02.170637 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0119fc2_7a0d_4f10_ae6a_e583297b08e4.slice/crio-28533ecacda9c813a12cdbf8efde7dcdb84fe57dce32ddcc367caffefd5e35d4 WatchSource:0}: Error finding container 28533ecacda9c813a12cdbf8efde7dcdb84fe57dce32ddcc367caffefd5e35d4: Status 404 returned error can't find the container with id 28533ecacda9c813a12cdbf8efde7dcdb84fe57dce32ddcc367caffefd5e35d4 Jul 27 21:05:02.826878 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:02.826833 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q9q9v" event={"ID":"f0119fc2-7a0d-4f10-ae6a-e583297b08e4","Type":"ContainerStarted","Data":"28533ecacda9c813a12cdbf8efde7dcdb84fe57dce32ddcc367caffefd5e35d4"} Jul 27 21:05:03.832096 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:03.832055 2577 generic.go:356] "Generic (PLEG): container finished" podID="f0119fc2-7a0d-4f10-ae6a-e583297b08e4" containerID="5d21cb933a1d57d75069fda1ff1e9fcf7af0bfca24523d7307ecd9291c87030f" exitCode=0 Jul 27 21:05:03.832523 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:03.832118 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q9q9v" event={"ID":"f0119fc2-7a0d-4f10-ae6a-e583297b08e4","Type":"ContainerDied","Data":"5d21cb933a1d57d75069fda1ff1e9fcf7af0bfca24523d7307ecd9291c87030f"} Jul 27 21:05:04.838269 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:04.838234 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q9q9v" event={"ID":"f0119fc2-7a0d-4f10-ae6a-e583297b08e4","Type":"ContainerStarted","Data":"054f9298315813c6d205981d5c79b8851e2b60927004bdb86213faa52198143a"} Jul 27 21:05:04.838712 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:04.838291 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q9q9v" event={"ID":"f0119fc2-7a0d-4f10-ae6a-e583297b08e4","Type":"ContainerStarted","Data":"7d9687af159038674ac3df0d7484decf8f21b4e17c0d3d5c90ca96e64c7378b9"} Jul 27 21:05:04.857551 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:04.857488 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-q9q9v" podStartSLOduration=2.581317089 podStartE2EDuration="3.857465602s" totalImagesPullingTime="1.20719934s" totalInitContainerRuntime="68.949173ms" isStatefulPod=true podCreationTimestamp="2026-07-27 21:05:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:05:04.856818845 +0000 UTC m=+96.988435079" watchObservedRunningTime="2026-07-27 21:05:04.857465602 +0000 UTC m=+96.989081825" Jul 27 21:05:05.689390 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:05.689353 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-lrnv5" Jul 27 21:05:06.897623 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:06.897565 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-75fb88d74f-jhdnw"] Jul 27 21:05:08.362803 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:08.362765 2577 scope.go:117] "RemoveContainer" containerID="4405d2e6c65d430a7fc45b6d010573fdb2b668d3a1670c71a5f26c8d00695a6c" Jul 27 21:05:08.855741 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:08.855709 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-j5rmn_4812c628-5f41-4e36-aaf6-8d62e4b6a443/console-operator/2.log" Jul 27 21:05:08.855918 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:08.855764 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-j5rmn" event={"ID":"4812c628-5f41-4e36-aaf6-8d62e4b6a443","Type":"ContainerStarted","Data":"25b26c864527d19f3f40049ccd52fbb388ef61a68a32270d44310d8b593b4224"} Jul 27 21:05:08.856055 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:08.856032 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-j5rmn" Jul 27 21:05:08.875564 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:08.875504 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-j5rmn" podStartSLOduration=49.696719585 podStartE2EDuration="55.875482752s" totalImagesPullingTime="6.178763167s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:04:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:05:08.874223741 +0000 UTC m=+101.005839958" watchObservedRunningTime="2026-07-27 21:05:08.875482752 +0000 UTC m=+101.007099020" Jul 27 21:05:09.032018 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:09.031989 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-j5rmn" Jul 27 21:05:09.216992 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:09.216909 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-fbz44"] Jul 27 21:05:09.220461 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:09.220435 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-fbz44" Jul 27 21:05:09.223016 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:09.222989 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 27 21:05:09.223130 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:09.223107 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 27 21:05:09.223192 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:09.223177 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-x6h5w\"" Jul 27 21:05:09.231619 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:09.231594 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-fbz44"] Jul 27 21:05:09.317689 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:09.317647 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4m9w\" (UniqueName: \"kubernetes.io/projected/145465f3-f266-413f-8a06-54e1db6e1f8a-kube-api-access-l4m9w\") pod \"downloads-7c97dc4555-fbz44\" (UID: \"145465f3-f266-413f-8a06-54e1db6e1f8a\") " pod="openshift-console/downloads-7c97dc4555-fbz44" Jul 27 21:05:09.418928 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:09.418886 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4m9w\" (UniqueName: \"kubernetes.io/projected/145465f3-f266-413f-8a06-54e1db6e1f8a-kube-api-access-l4m9w\") pod \"downloads-7c97dc4555-fbz44\" (UID: \"145465f3-f266-413f-8a06-54e1db6e1f8a\") " pod="openshift-console/downloads-7c97dc4555-fbz44" Jul 27 21:05:09.427190 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:09.427161 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4m9w\" (UniqueName: \"kubernetes.io/projected/145465f3-f266-413f-8a06-54e1db6e1f8a-kube-api-access-l4m9w\") pod \"downloads-7c97dc4555-fbz44\" (UID: \"145465f3-f266-413f-8a06-54e1db6e1f8a\") " pod="openshift-console/downloads-7c97dc4555-fbz44" Jul 27 21:05:09.530443 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:09.530402 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-fbz44" Jul 27 21:05:09.667118 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:09.667087 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-fbz44"] Jul 27 21:05:09.671014 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:05:09.670970 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod145465f3_f266_413f_8a06_54e1db6e1f8a.slice/crio-f1579cca12fd9ac4dba1e11cf49213f685044b35a0a93caaf28ac3db9c95a60a WatchSource:0}: Error finding container f1579cca12fd9ac4dba1e11cf49213f685044b35a0a93caaf28ac3db9c95a60a: Status 404 returned error can't find the container with id f1579cca12fd9ac4dba1e11cf49213f685044b35a0a93caaf28ac3db9c95a60a Jul 27 21:05:09.861229 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:09.861144 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-fbz44" event={"ID":"145465f3-f266-413f-8a06-54e1db6e1f8a","Type":"ContainerStarted","Data":"f1579cca12fd9ac4dba1e11cf49213f685044b35a0a93caaf28ac3db9c95a60a"} Jul 27 21:05:15.776928 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:15.776884 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-85b49db84c-gv84n"] Jul 27 21:05:15.948751 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:15.948707 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85b49db84c-gv84n"] Jul 27 21:05:15.948945 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:15.948855 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85b49db84c-gv84n" Jul 27 21:05:15.951648 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:15.951619 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 27 21:05:15.951869 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:15.951848 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 27 21:05:15.952012 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:15.951910 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 27 21:05:15.952108 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:15.952086 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 27 21:05:15.953074 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:15.952876 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 27 21:05:15.953074 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:15.952935 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-lbj4g\"" Jul 27 21:05:16.081815 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:16.081733 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d5d319e-38b7-4c57-8b26-262f7ce01b2b-console-serving-cert\") pod \"console-85b49db84c-gv84n\" (UID: \"1d5d319e-38b7-4c57-8b26-262f7ce01b2b\") " pod="openshift-console/console-85b49db84c-gv84n" Jul 27 21:05:16.081815 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:16.081780 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzflr\" (UniqueName: \"kubernetes.io/projected/1d5d319e-38b7-4c57-8b26-262f7ce01b2b-kube-api-access-zzflr\") pod \"console-85b49db84c-gv84n\" (UID: \"1d5d319e-38b7-4c57-8b26-262f7ce01b2b\") " pod="openshift-console/console-85b49db84c-gv84n" Jul 27 21:05:16.081815 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:16.081811 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1d5d319e-38b7-4c57-8b26-262f7ce01b2b-console-config\") pod \"console-85b49db84c-gv84n\" (UID: \"1d5d319e-38b7-4c57-8b26-262f7ce01b2b\") " pod="openshift-console/console-85b49db84c-gv84n" Jul 27 21:05:16.082097 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:16.081905 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d5d319e-38b7-4c57-8b26-262f7ce01b2b-oauth-serving-cert\") pod \"console-85b49db84c-gv84n\" (UID: \"1d5d319e-38b7-4c57-8b26-262f7ce01b2b\") " pod="openshift-console/console-85b49db84c-gv84n" Jul 27 21:05:16.082097 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:16.081945 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d5d319e-38b7-4c57-8b26-262f7ce01b2b-console-oauth-config\") pod \"console-85b49db84c-gv84n\" (UID: \"1d5d319e-38b7-4c57-8b26-262f7ce01b2b\") " pod="openshift-console/console-85b49db84c-gv84n" Jul 27 21:05:16.082097 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:16.081971 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1d5d319e-38b7-4c57-8b26-262f7ce01b2b-service-ca\") pod \"console-85b49db84c-gv84n\" (UID: \"1d5d319e-38b7-4c57-8b26-262f7ce01b2b\") " pod="openshift-console/console-85b49db84c-gv84n" Jul 27 21:05:16.182801 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:16.182707 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzflr\" (UniqueName: \"kubernetes.io/projected/1d5d319e-38b7-4c57-8b26-262f7ce01b2b-kube-api-access-zzflr\") pod \"console-85b49db84c-gv84n\" (UID: \"1d5d319e-38b7-4c57-8b26-262f7ce01b2b\") " pod="openshift-console/console-85b49db84c-gv84n" Jul 27 21:05:16.183018 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:16.182920 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1d5d319e-38b7-4c57-8b26-262f7ce01b2b-console-config\") pod \"console-85b49db84c-gv84n\" (UID: \"1d5d319e-38b7-4c57-8b26-262f7ce01b2b\") " pod="openshift-console/console-85b49db84c-gv84n" Jul 27 21:05:16.183018 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:16.182981 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d5d319e-38b7-4c57-8b26-262f7ce01b2b-oauth-serving-cert\") pod \"console-85b49db84c-gv84n\" (UID: \"1d5d319e-38b7-4c57-8b26-262f7ce01b2b\") " pod="openshift-console/console-85b49db84c-gv84n" Jul 27 21:05:16.183139 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:16.183015 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d5d319e-38b7-4c57-8b26-262f7ce01b2b-console-oauth-config\") pod \"console-85b49db84c-gv84n\" (UID: \"1d5d319e-38b7-4c57-8b26-262f7ce01b2b\") " pod="openshift-console/console-85b49db84c-gv84n" Jul 27 21:05:16.183139 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:16.183044 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1d5d319e-38b7-4c57-8b26-262f7ce01b2b-service-ca\") pod \"console-85b49db84c-gv84n\" (UID: \"1d5d319e-38b7-4c57-8b26-262f7ce01b2b\") " pod="openshift-console/console-85b49db84c-gv84n" Jul 27 21:05:16.183235 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:16.183179 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d5d319e-38b7-4c57-8b26-262f7ce01b2b-console-serving-cert\") pod \"console-85b49db84c-gv84n\" (UID: \"1d5d319e-38b7-4c57-8b26-262f7ce01b2b\") " pod="openshift-console/console-85b49db84c-gv84n" Jul 27 21:05:16.183884 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:16.183804 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1d5d319e-38b7-4c57-8b26-262f7ce01b2b-console-config\") pod \"console-85b49db84c-gv84n\" (UID: \"1d5d319e-38b7-4c57-8b26-262f7ce01b2b\") " pod="openshift-console/console-85b49db84c-gv84n" Jul 27 21:05:16.183884 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:16.183848 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1d5d319e-38b7-4c57-8b26-262f7ce01b2b-service-ca\") pod \"console-85b49db84c-gv84n\" (UID: \"1d5d319e-38b7-4c57-8b26-262f7ce01b2b\") " pod="openshift-console/console-85b49db84c-gv84n" Jul 27 21:05:16.184061 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:16.183884 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d5d319e-38b7-4c57-8b26-262f7ce01b2b-oauth-serving-cert\") pod \"console-85b49db84c-gv84n\" (UID: \"1d5d319e-38b7-4c57-8b26-262f7ce01b2b\") " pod="openshift-console/console-85b49db84c-gv84n" Jul 27 21:05:16.185753 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:16.185732 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d5d319e-38b7-4c57-8b26-262f7ce01b2b-console-oauth-config\") pod \"console-85b49db84c-gv84n\" (UID: \"1d5d319e-38b7-4c57-8b26-262f7ce01b2b\") " pod="openshift-console/console-85b49db84c-gv84n" Jul 27 21:05:16.186036 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:16.186016 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d5d319e-38b7-4c57-8b26-262f7ce01b2b-console-serving-cert\") pod \"console-85b49db84c-gv84n\" (UID: \"1d5d319e-38b7-4c57-8b26-262f7ce01b2b\") " pod="openshift-console/console-85b49db84c-gv84n" Jul 27 21:05:16.191945 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:16.191906 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzflr\" (UniqueName: \"kubernetes.io/projected/1d5d319e-38b7-4c57-8b26-262f7ce01b2b-kube-api-access-zzflr\") pod \"console-85b49db84c-gv84n\" (UID: \"1d5d319e-38b7-4c57-8b26-262f7ce01b2b\") " pod="openshift-console/console-85b49db84c-gv84n" Jul 27 21:05:16.260955 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:16.260895 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85b49db84c-gv84n" Jul 27 21:05:16.462801 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:16.462698 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85b49db84c-gv84n"] Jul 27 21:05:16.466777 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:05:16.466742 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d5d319e_38b7_4c57_8b26_262f7ce01b2b.slice/crio-6438ae0da76edcfc6c6f8cb04aab5295040fd3f931e7ae72fcce3d226696f13f WatchSource:0}: Error finding container 6438ae0da76edcfc6c6f8cb04aab5295040fd3f931e7ae72fcce3d226696f13f: Status 404 returned error can't find the container with id 6438ae0da76edcfc6c6f8cb04aab5295040fd3f931e7ae72fcce3d226696f13f Jul 27 21:05:16.891334 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:16.891297 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85b49db84c-gv84n" event={"ID":"1d5d319e-38b7-4c57-8b26-262f7ce01b2b","Type":"ContainerStarted","Data":"6438ae0da76edcfc6c6f8cb04aab5295040fd3f931e7ae72fcce3d226696f13f"} Jul 27 21:05:17.734617 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:17.734203 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-776bb4b4b8-djl4p"] Jul 27 21:05:17.797629 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:17.797591 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-776bb4b4b8-djl4p"] Jul 27 21:05:17.797829 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:17.797713 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-776bb4b4b8-djl4p" Jul 27 21:05:17.806137 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:17.806106 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 27 21:05:17.900783 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:17.900745 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0b07e3ce-07f7-4df9-8d82-d58cf7a3b285-console-serving-cert\") pod \"console-776bb4b4b8-djl4p\" (UID: \"0b07e3ce-07f7-4df9-8d82-d58cf7a3b285\") " pod="openshift-console/console-776bb4b4b8-djl4p" Jul 27 21:05:17.901247 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:17.900811 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b07e3ce-07f7-4df9-8d82-d58cf7a3b285-trusted-ca-bundle\") pod \"console-776bb4b4b8-djl4p\" (UID: \"0b07e3ce-07f7-4df9-8d82-d58cf7a3b285\") " pod="openshift-console/console-776bb4b4b8-djl4p" Jul 27 21:05:17.901247 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:17.900841 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0b07e3ce-07f7-4df9-8d82-d58cf7a3b285-oauth-serving-cert\") pod \"console-776bb4b4b8-djl4p\" (UID: \"0b07e3ce-07f7-4df9-8d82-d58cf7a3b285\") " pod="openshift-console/console-776bb4b4b8-djl4p" Jul 27 21:05:17.901247 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:17.900884 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm4pd\" (UniqueName: \"kubernetes.io/projected/0b07e3ce-07f7-4df9-8d82-d58cf7a3b285-kube-api-access-hm4pd\") pod \"console-776bb4b4b8-djl4p\" (UID: \"0b07e3ce-07f7-4df9-8d82-d58cf7a3b285\") " pod="openshift-console/console-776bb4b4b8-djl4p" Jul 27 21:05:17.901247 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:17.900919 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0b07e3ce-07f7-4df9-8d82-d58cf7a3b285-console-config\") pod \"console-776bb4b4b8-djl4p\" (UID: \"0b07e3ce-07f7-4df9-8d82-d58cf7a3b285\") " pod="openshift-console/console-776bb4b4b8-djl4p" Jul 27 21:05:17.901247 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:17.900964 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0b07e3ce-07f7-4df9-8d82-d58cf7a3b285-console-oauth-config\") pod \"console-776bb4b4b8-djl4p\" (UID: \"0b07e3ce-07f7-4df9-8d82-d58cf7a3b285\") " pod="openshift-console/console-776bb4b4b8-djl4p" Jul 27 21:05:17.901247 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:17.901020 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0b07e3ce-07f7-4df9-8d82-d58cf7a3b285-service-ca\") pod \"console-776bb4b4b8-djl4p\" (UID: \"0b07e3ce-07f7-4df9-8d82-d58cf7a3b285\") " pod="openshift-console/console-776bb4b4b8-djl4p" Jul 27 21:05:18.001982 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:18.001898 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0b07e3ce-07f7-4df9-8d82-d58cf7a3b285-oauth-serving-cert\") pod \"console-776bb4b4b8-djl4p\" (UID: \"0b07e3ce-07f7-4df9-8d82-d58cf7a3b285\") " pod="openshift-console/console-776bb4b4b8-djl4p" Jul 27 21:05:18.001982 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:18.001964 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm4pd\" (UniqueName: \"kubernetes.io/projected/0b07e3ce-07f7-4df9-8d82-d58cf7a3b285-kube-api-access-hm4pd\") pod \"console-776bb4b4b8-djl4p\" (UID: \"0b07e3ce-07f7-4df9-8d82-d58cf7a3b285\") " pod="openshift-console/console-776bb4b4b8-djl4p" Jul 27 21:05:18.002215 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:18.002002 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0b07e3ce-07f7-4df9-8d82-d58cf7a3b285-console-config\") pod \"console-776bb4b4b8-djl4p\" (UID: \"0b07e3ce-07f7-4df9-8d82-d58cf7a3b285\") " pod="openshift-console/console-776bb4b4b8-djl4p" Jul 27 21:05:18.002215 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:18.002047 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0b07e3ce-07f7-4df9-8d82-d58cf7a3b285-console-oauth-config\") pod \"console-776bb4b4b8-djl4p\" (UID: \"0b07e3ce-07f7-4df9-8d82-d58cf7a3b285\") " pod="openshift-console/console-776bb4b4b8-djl4p" Jul 27 21:05:18.002215 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:18.002095 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0b07e3ce-07f7-4df9-8d82-d58cf7a3b285-service-ca\") pod \"console-776bb4b4b8-djl4p\" (UID: \"0b07e3ce-07f7-4df9-8d82-d58cf7a3b285\") " pod="openshift-console/console-776bb4b4b8-djl4p" Jul 27 21:05:18.002215 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:18.002146 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0b07e3ce-07f7-4df9-8d82-d58cf7a3b285-console-serving-cert\") pod \"console-776bb4b4b8-djl4p\" (UID: \"0b07e3ce-07f7-4df9-8d82-d58cf7a3b285\") " pod="openshift-console/console-776bb4b4b8-djl4p" Jul 27 21:05:18.002215 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:18.002185 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b07e3ce-07f7-4df9-8d82-d58cf7a3b285-trusted-ca-bundle\") pod \"console-776bb4b4b8-djl4p\" (UID: \"0b07e3ce-07f7-4df9-8d82-d58cf7a3b285\") " pod="openshift-console/console-776bb4b4b8-djl4p" Jul 27 21:05:18.003174 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:18.003144 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0b07e3ce-07f7-4df9-8d82-d58cf7a3b285-oauth-serving-cert\") pod \"console-776bb4b4b8-djl4p\" (UID: \"0b07e3ce-07f7-4df9-8d82-d58cf7a3b285\") " pod="openshift-console/console-776bb4b4b8-djl4p" Jul 27 21:05:18.003291 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:18.003223 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0b07e3ce-07f7-4df9-8d82-d58cf7a3b285-console-config\") pod \"console-776bb4b4b8-djl4p\" (UID: \"0b07e3ce-07f7-4df9-8d82-d58cf7a3b285\") " pod="openshift-console/console-776bb4b4b8-djl4p" Jul 27 21:05:18.003417 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:18.003365 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0b07e3ce-07f7-4df9-8d82-d58cf7a3b285-service-ca\") pod \"console-776bb4b4b8-djl4p\" (UID: \"0b07e3ce-07f7-4df9-8d82-d58cf7a3b285\") " pod="openshift-console/console-776bb4b4b8-djl4p" Jul 27 21:05:18.004010 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:18.003982 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b07e3ce-07f7-4df9-8d82-d58cf7a3b285-trusted-ca-bundle\") pod \"console-776bb4b4b8-djl4p\" (UID: \"0b07e3ce-07f7-4df9-8d82-d58cf7a3b285\") " pod="openshift-console/console-776bb4b4b8-djl4p" Jul 27 21:05:18.005405 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:18.005379 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0b07e3ce-07f7-4df9-8d82-d58cf7a3b285-console-serving-cert\") pod \"console-776bb4b4b8-djl4p\" (UID: \"0b07e3ce-07f7-4df9-8d82-d58cf7a3b285\") " pod="openshift-console/console-776bb4b4b8-djl4p" Jul 27 21:05:18.005518 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:18.005466 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0b07e3ce-07f7-4df9-8d82-d58cf7a3b285-console-oauth-config\") pod \"console-776bb4b4b8-djl4p\" (UID: \"0b07e3ce-07f7-4df9-8d82-d58cf7a3b285\") " pod="openshift-console/console-776bb4b4b8-djl4p" Jul 27 21:05:18.011907 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:18.011870 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm4pd\" (UniqueName: \"kubernetes.io/projected/0b07e3ce-07f7-4df9-8d82-d58cf7a3b285-kube-api-access-hm4pd\") pod \"console-776bb4b4b8-djl4p\" (UID: \"0b07e3ce-07f7-4df9-8d82-d58cf7a3b285\") " pod="openshift-console/console-776bb4b4b8-djl4p" Jul 27 21:05:18.111765 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:18.111472 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-776bb4b4b8-djl4p" Jul 27 21:05:18.282549 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:18.282450 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-776bb4b4b8-djl4p"] Jul 27 21:05:18.286872 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:05:18.286834 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b07e3ce_07f7_4df9_8d82_d58cf7a3b285.slice/crio-a37ef76959eeb996a28cf79943a0f73bc27d0b507063fdd89432c672cf7d3c05 WatchSource:0}: Error finding container a37ef76959eeb996a28cf79943a0f73bc27d0b507063fdd89432c672cf7d3c05: Status 404 returned error can't find the container with id a37ef76959eeb996a28cf79943a0f73bc27d0b507063fdd89432c672cf7d3c05 Jul 27 21:05:18.902435 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:18.902387 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-776bb4b4b8-djl4p" event={"ID":"0b07e3ce-07f7-4df9-8d82-d58cf7a3b285","Type":"ContainerStarted","Data":"a37ef76959eeb996a28cf79943a0f73bc27d0b507063fdd89432c672cf7d3c05"} Jul 27 21:05:26.944538 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:26.944447 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-fbz44" event={"ID":"145465f3-f266-413f-8a06-54e1db6e1f8a","Type":"ContainerStarted","Data":"f363eeadef3435cf28065cdc909b85f11ab5657cbc750a5c8cd25d7ce2f40ade"} Jul 27 21:05:26.945004 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:26.944604 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-fbz44" Jul 27 21:05:26.946637 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:26.946605 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85b49db84c-gv84n" event={"ID":"1d5d319e-38b7-4c57-8b26-262f7ce01b2b","Type":"ContainerStarted","Data":"fc4b9ef120820ea7bde15c1eaa51b5a299d931f97997f48840e0cdd13c0764fd"} Jul 27 21:05:26.948503 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:26.948471 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-776bb4b4b8-djl4p" event={"ID":"0b07e3ce-07f7-4df9-8d82-d58cf7a3b285","Type":"ContainerStarted","Data":"64151aadf9c9f46cdba8a9fca35a7ac77ecec43c28ef9de8ef44171047b1a4e8"} Jul 27 21:05:26.958523 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:26.958487 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-fbz44" Jul 27 21:05:26.965424 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:26.965360 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-fbz44" podStartSLOduration=0.940268295 podStartE2EDuration="17.965343224s" totalImagesPullingTime="17.025074929s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:05:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:05:26.964556495 +0000 UTC m=+119.096172717" watchObservedRunningTime="2026-07-27 21:05:26.965343224 +0000 UTC m=+119.096959446" Jul 27 21:05:26.984630 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:26.984540 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-776bb4b4b8-djl4p" podStartSLOduration=1.615766899 podStartE2EDuration="9.984521185s" totalImagesPullingTime="8.368754286s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:05:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:05:26.983173606 +0000 UTC m=+119.114789840" watchObservedRunningTime="2026-07-27 21:05:26.984521185 +0000 UTC m=+119.116137408" Jul 27 21:05:27.021127 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:27.021065 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-85b49db84c-gv84n" podStartSLOduration=1.83305855 podStartE2EDuration="12.021039295s" totalImagesPullingTime="10.187980745s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:05:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:05:27.01884793 +0000 UTC m=+119.150464179" watchObservedRunningTime="2026-07-27 21:05:27.021039295 +0000 UTC m=+119.152655519" Jul 27 21:05:28.111809 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:28.111766 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-776bb4b4b8-djl4p" Jul 27 21:05:28.112295 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:28.111843 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-776bb4b4b8-djl4p" Jul 27 21:05:28.117753 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:28.117716 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-776bb4b4b8-djl4p" Jul 27 21:05:28.963082 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:28.963039 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-776bb4b4b8-djl4p" Jul 27 21:05:29.018694 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:29.018658 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85b49db84c-gv84n"] Jul 27 21:05:31.921200 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:31.921146 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-75fb88d74f-jhdnw" podUID="eb9ca170-7ea1-45de-aca4-54b4731ed82d" containerName="registry" containerID="cri-o://ac2ec9993d1aab75ccd7a17254aea313a695d570933ebc61805e6d5f1d135581" gracePeriod=30 Jul 27 21:05:32.240029 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:32.239986 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75fb88d74f-jhdnw" Jul 27 21:05:32.340856 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:32.340812 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eb9ca170-7ea1-45de-aca4-54b4731ed82d-bound-sa-token\") pod \"eb9ca170-7ea1-45de-aca4-54b4731ed82d\" (UID: \"eb9ca170-7ea1-45de-aca4-54b4731ed82d\") " Jul 27 21:05:32.341151 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:32.340877 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/eb9ca170-7ea1-45de-aca4-54b4731ed82d-ca-trust-extracted\") pod \"eb9ca170-7ea1-45de-aca4-54b4731ed82d\" (UID: \"eb9ca170-7ea1-45de-aca4-54b4731ed82d\") " Jul 27 21:05:32.341151 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:32.340945 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/eb9ca170-7ea1-45de-aca4-54b4731ed82d-registry-certificates\") pod \"eb9ca170-7ea1-45de-aca4-54b4731ed82d\" (UID: \"eb9ca170-7ea1-45de-aca4-54b4731ed82d\") " Jul 27 21:05:32.341151 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:32.340979 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eb9ca170-7ea1-45de-aca4-54b4731ed82d-trusted-ca\") pod \"eb9ca170-7ea1-45de-aca4-54b4731ed82d\" (UID: \"eb9ca170-7ea1-45de-aca4-54b4731ed82d\") " Jul 27 21:05:32.341151 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:32.341005 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/eb9ca170-7ea1-45de-aca4-54b4731ed82d-installation-pull-secrets\") pod \"eb9ca170-7ea1-45de-aca4-54b4731ed82d\" (UID: \"eb9ca170-7ea1-45de-aca4-54b4731ed82d\") " Jul 27 21:05:32.341151 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:32.341044 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/eb9ca170-7ea1-45de-aca4-54b4731ed82d-image-registry-private-configuration\") pod \"eb9ca170-7ea1-45de-aca4-54b4731ed82d\" (UID: \"eb9ca170-7ea1-45de-aca4-54b4731ed82d\") " Jul 27 21:05:32.341151 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:32.341080 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eb9ca170-7ea1-45de-aca4-54b4731ed82d-registry-tls\") pod \"eb9ca170-7ea1-45de-aca4-54b4731ed82d\" (UID: \"eb9ca170-7ea1-45de-aca4-54b4731ed82d\") " Jul 27 21:05:32.341151 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:32.341143 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s62d6\" (UniqueName: \"kubernetes.io/projected/eb9ca170-7ea1-45de-aca4-54b4731ed82d-kube-api-access-s62d6\") pod \"eb9ca170-7ea1-45de-aca4-54b4731ed82d\" (UID: \"eb9ca170-7ea1-45de-aca4-54b4731ed82d\") " Jul 27 21:05:32.341602 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:32.341347 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb9ca170-7ea1-45de-aca4-54b4731ed82d-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "eb9ca170-7ea1-45de-aca4-54b4731ed82d" (UID: "eb9ca170-7ea1-45de-aca4-54b4731ed82d"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 21:05:32.341602 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:32.341441 2577 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/eb9ca170-7ea1-45de-aca4-54b4731ed82d-registry-certificates\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\"" Jul 27 21:05:32.341712 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:32.341663 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb9ca170-7ea1-45de-aca4-54b4731ed82d-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "eb9ca170-7ea1-45de-aca4-54b4731ed82d" (UID: "eb9ca170-7ea1-45de-aca4-54b4731ed82d"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 21:05:32.344951 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:32.344882 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb9ca170-7ea1-45de-aca4-54b4731ed82d-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "eb9ca170-7ea1-45de-aca4-54b4731ed82d" (UID: "eb9ca170-7ea1-45de-aca4-54b4731ed82d"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 21:05:32.345254 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:32.345133 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb9ca170-7ea1-45de-aca4-54b4731ed82d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "eb9ca170-7ea1-45de-aca4-54b4731ed82d" (UID: "eb9ca170-7ea1-45de-aca4-54b4731ed82d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 21:05:32.345667 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:32.345612 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb9ca170-7ea1-45de-aca4-54b4731ed82d-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "eb9ca170-7ea1-45de-aca4-54b4731ed82d" (UID: "eb9ca170-7ea1-45de-aca4-54b4731ed82d"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 21:05:32.345779 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:32.345696 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb9ca170-7ea1-45de-aca4-54b4731ed82d-kube-api-access-s62d6" (OuterVolumeSpecName: "kube-api-access-s62d6") pod "eb9ca170-7ea1-45de-aca4-54b4731ed82d" (UID: "eb9ca170-7ea1-45de-aca4-54b4731ed82d"). InnerVolumeSpecName "kube-api-access-s62d6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 21:05:32.345841 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:32.345792 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb9ca170-7ea1-45de-aca4-54b4731ed82d-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "eb9ca170-7ea1-45de-aca4-54b4731ed82d" (UID: "eb9ca170-7ea1-45de-aca4-54b4731ed82d"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 21:05:32.353843 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:32.353799 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb9ca170-7ea1-45de-aca4-54b4731ed82d-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "eb9ca170-7ea1-45de-aca4-54b4731ed82d" (UID: "eb9ca170-7ea1-45de-aca4-54b4731ed82d"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 21:05:32.442740 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:32.442697 2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eb9ca170-7ea1-45de-aca4-54b4731ed82d-trusted-ca\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\"" Jul 27 21:05:32.442740 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:32.442729 2577 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/eb9ca170-7ea1-45de-aca4-54b4731ed82d-installation-pull-secrets\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\"" Jul 27 21:05:32.442740 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:32.442745 2577 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/eb9ca170-7ea1-45de-aca4-54b4731ed82d-image-registry-private-configuration\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\"" Jul 27 21:05:32.443042 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:32.442759 2577 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eb9ca170-7ea1-45de-aca4-54b4731ed82d-registry-tls\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\"" Jul 27 21:05:32.443042 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:32.442774 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s62d6\" (UniqueName: \"kubernetes.io/projected/eb9ca170-7ea1-45de-aca4-54b4731ed82d-kube-api-access-s62d6\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\"" Jul 27 21:05:32.443042 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:32.442786 2577 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eb9ca170-7ea1-45de-aca4-54b4731ed82d-bound-sa-token\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\"" Jul 27 21:05:32.443042 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:32.442799 2577 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/eb9ca170-7ea1-45de-aca4-54b4731ed82d-ca-trust-extracted\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\"" Jul 27 21:05:32.976750 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:32.976709 2577 generic.go:356] "Generic (PLEG): container finished" podID="eb9ca170-7ea1-45de-aca4-54b4731ed82d" containerID="ac2ec9993d1aab75ccd7a17254aea313a695d570933ebc61805e6d5f1d135581" exitCode=0 Jul 27 21:05:32.977330 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:32.976759 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75fb88d74f-jhdnw" Jul 27 21:05:32.977330 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:32.976763 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75fb88d74f-jhdnw" event={"ID":"eb9ca170-7ea1-45de-aca4-54b4731ed82d","Type":"ContainerDied","Data":"ac2ec9993d1aab75ccd7a17254aea313a695d570933ebc61805e6d5f1d135581"} Jul 27 21:05:32.977330 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:32.976815 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75fb88d74f-jhdnw" event={"ID":"eb9ca170-7ea1-45de-aca4-54b4731ed82d","Type":"ContainerDied","Data":"69b1c381a6c137daa757306a9262bcca7b1b5f9d8cc474676398dd5f8b2e7b49"} Jul 27 21:05:32.977330 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:32.976843 2577 scope.go:117] "RemoveContainer" containerID="ac2ec9993d1aab75ccd7a17254aea313a695d570933ebc61805e6d5f1d135581" Jul 27 21:05:32.987445 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:32.987415 2577 scope.go:117] "RemoveContainer" containerID="ac2ec9993d1aab75ccd7a17254aea313a695d570933ebc61805e6d5f1d135581" Jul 27 21:05:32.987872 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:05:32.987842 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac2ec9993d1aab75ccd7a17254aea313a695d570933ebc61805e6d5f1d135581\": container with ID starting with ac2ec9993d1aab75ccd7a17254aea313a695d570933ebc61805e6d5f1d135581 not found: ID does not exist" containerID="ac2ec9993d1aab75ccd7a17254aea313a695d570933ebc61805e6d5f1d135581" Jul 27 21:05:32.987962 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:32.987884 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac2ec9993d1aab75ccd7a17254aea313a695d570933ebc61805e6d5f1d135581"} err="failed to get container status \"ac2ec9993d1aab75ccd7a17254aea313a695d570933ebc61805e6d5f1d135581\": rpc error: code = NotFound desc = could not find container \"ac2ec9993d1aab75ccd7a17254aea313a695d570933ebc61805e6d5f1d135581\": container with ID starting with ac2ec9993d1aab75ccd7a17254aea313a695d570933ebc61805e6d5f1d135581 not found: ID does not exist" Jul 27 21:05:32.995921 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:32.995876 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-75fb88d74f-jhdnw"] Jul 27 21:05:33.002543 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:33.002504 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-75fb88d74f-jhdnw"] Jul 27 21:05:34.366765 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:34.366724 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb9ca170-7ea1-45de-aca4-54b4731ed82d" path="/var/lib/kubelet/pods/eb9ca170-7ea1-45de-aca4-54b4731ed82d/volumes" Jul 27 21:05:36.262010 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:36.261972 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-85b49db84c-gv84n" Jul 27 21:05:38.188340 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:38.188302 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c32154a-b41d-45c4-afa6-240a8ea24835-metrics-certs\") pod \"network-metrics-daemon-lvg8z\" (UID: \"0c32154a-b41d-45c4-afa6-240a8ea24835\") " pod="openshift-multus/network-metrics-daemon-lvg8z" Jul 27 21:05:38.190617 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:38.190585 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c32154a-b41d-45c4-afa6-240a8ea24835-metrics-certs\") pod \"network-metrics-daemon-lvg8z\" (UID: \"0c32154a-b41d-45c4-afa6-240a8ea24835\") " pod="openshift-multus/network-metrics-daemon-lvg8z" Jul 27 21:05:38.286862 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:38.286827 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bztw5\"" Jul 27 21:05:38.294184 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:38.294159 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lvg8z" Jul 27 21:05:38.491741 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:38.491707 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lvg8z"] Jul 27 21:05:38.495365 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:05:38.495331 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c32154a_b41d_45c4_afa6_240a8ea24835.slice/crio-c6f93af46eafd7542a7ba9a9c4b8c0ffe0669815ce911160e56de7eeef05b74b WatchSource:0}: Error finding container c6f93af46eafd7542a7ba9a9c4b8c0ffe0669815ce911160e56de7eeef05b74b: Status 404 returned error can't find the container with id c6f93af46eafd7542a7ba9a9c4b8c0ffe0669815ce911160e56de7eeef05b74b Jul 27 21:05:39.003533 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:39.003496 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lvg8z" event={"ID":"0c32154a-b41d-45c4-afa6-240a8ea24835","Type":"ContainerStarted","Data":"c6f93af46eafd7542a7ba9a9c4b8c0ffe0669815ce911160e56de7eeef05b74b"} Jul 27 21:05:41.013244 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:41.013211 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lvg8z" event={"ID":"0c32154a-b41d-45c4-afa6-240a8ea24835","Type":"ContainerStarted","Data":"12c170eadcb25cfbe5a01cd9440e819a979b63e5a89b24a66e384d95c54d6227"} Jul 27 21:05:41.013753 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:41.013272 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lvg8z" event={"ID":"0c32154a-b41d-45c4-afa6-240a8ea24835","Type":"ContainerStarted","Data":"381186169f41729233502329903d0f9dfb499957b5c1dcb9780ee02f3161cb27"} Jul 27 21:05:41.014747 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:41.014723 2577 generic.go:356] "Generic (PLEG): container finished" podID="b2e7d196-079c-4bcd-aaf0-affbae8a0594" containerID="eaa0a079061ba39109b685264deb4a8e6b352bfff4e5eb755d4edf013495a517" exitCode=0 Jul 27 21:05:41.014877 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:41.014802 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-9rp9b" event={"ID":"b2e7d196-079c-4bcd-aaf0-affbae8a0594","Type":"ContainerDied","Data":"eaa0a079061ba39109b685264deb4a8e6b352bfff4e5eb755d4edf013495a517"} Jul 27 21:05:41.015138 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:41.015121 2577 scope.go:117] "RemoveContainer" containerID="eaa0a079061ba39109b685264deb4a8e6b352bfff4e5eb755d4edf013495a517" Jul 27 21:05:41.061287 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:41.061239 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-lvg8z" podStartSLOduration=130.586987795 podStartE2EDuration="2m12.061221252s" totalImagesPullingTime="1.474233457s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:03:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:05:41.061000653 +0000 UTC m=+133.192616876" watchObservedRunningTime="2026-07-27 21:05:41.061221252 +0000 UTC m=+133.192837473" Jul 27 21:05:42.020906 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:42.020867 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-9rp9b" event={"ID":"b2e7d196-079c-4bcd-aaf0-affbae8a0594","Type":"ContainerStarted","Data":"55d3884b97609ce13e77fa9adf0afe3fcca2b5abde153ac695b3d0a80b992fb1"} Jul 27 21:05:46.039090 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:46.039053 2577 generic.go:356] "Generic (PLEG): container finished" podID="f33ee287-cfc5-436d-93cd-13801c2c522b" containerID="0ab0db102c864826e0a3119779c9d5030b1d639080af9576478ebb0fc79c30e6" exitCode=0 Jul 27 21:05:46.039647 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:46.039134 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-kwqx9" event={"ID":"f33ee287-cfc5-436d-93cd-13801c2c522b","Type":"ContainerDied","Data":"0ab0db102c864826e0a3119779c9d5030b1d639080af9576478ebb0fc79c30e6"} Jul 27 21:05:46.039647 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:46.039531 2577 scope.go:117] "RemoveContainer" containerID="0ab0db102c864826e0a3119779c9d5030b1d639080af9576478ebb0fc79c30e6" Jul 27 21:05:46.040958 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:46.040930 2577 generic.go:356] "Generic (PLEG): container finished" podID="79a93222-a9cb-4baf-8113-8e46390ff478" containerID="6e82f7d7696e085ebc94a01f0bf6b61edc6eedbb07708387ae2eb25dd8c1cb37" exitCode=0 Jul 27 21:05:46.041084 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:46.041000 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-trwhz" event={"ID":"79a93222-a9cb-4baf-8113-8e46390ff478","Type":"ContainerDied","Data":"6e82f7d7696e085ebc94a01f0bf6b61edc6eedbb07708387ae2eb25dd8c1cb37"} Jul 27 21:05:46.041329 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:46.041316 2577 scope.go:117] "RemoveContainer" containerID="6e82f7d7696e085ebc94a01f0bf6b61edc6eedbb07708387ae2eb25dd8c1cb37" Jul 27 21:05:46.734843 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:46.734811 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-64c559777f-75n27_86e4b5a1-4521-43b3-9bf8-c42ec92d21f8/router/0.log" Jul 27 21:05:46.743713 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:46.743684 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7gnng_597ea753-2547-4a21-9678-89da32128dcb/serve-healthcheck-canary/0.log" Jul 27 21:05:47.046462 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:47.046357 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-trwhz" event={"ID":"79a93222-a9cb-4baf-8113-8e46390ff478","Type":"ContainerStarted","Data":"bfa9369c384b1f04b778b26168f41fd3234ce138a906a7feb27985cf7e2fafb9"} Jul 27 21:05:47.048229 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:47.048207 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-kwqx9" event={"ID":"f33ee287-cfc5-436d-93cd-13801c2c522b","Type":"ContainerStarted","Data":"362f6898f3416c2c02197f369c44fafd7ab1ef7c0d6a89c7377b671ed4917715"} Jul 27 21:05:54.045016 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:54.044940 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-85b49db84c-gv84n" podUID="1d5d319e-38b7-4c57-8b26-262f7ce01b2b" containerName="console" containerID="cri-o://fc4b9ef120820ea7bde15c1eaa51b5a299d931f97997f48840e0cdd13c0764fd" gracePeriod=15 Jul 27 21:05:54.324490 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:54.324400 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85b49db84c-gv84n" Jul 27 21:05:54.428309 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:54.428270 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d5d319e-38b7-4c57-8b26-262f7ce01b2b-oauth-serving-cert\") pod \"1d5d319e-38b7-4c57-8b26-262f7ce01b2b\" (UID: \"1d5d319e-38b7-4c57-8b26-262f7ce01b2b\") " Jul 27 21:05:54.428504 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:54.428323 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d5d319e-38b7-4c57-8b26-262f7ce01b2b-console-oauth-config\") pod \"1d5d319e-38b7-4c57-8b26-262f7ce01b2b\" (UID: \"1d5d319e-38b7-4c57-8b26-262f7ce01b2b\") " Jul 27 21:05:54.428504 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:54.428364 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1d5d319e-38b7-4c57-8b26-262f7ce01b2b-console-config\") pod \"1d5d319e-38b7-4c57-8b26-262f7ce01b2b\" (UID: \"1d5d319e-38b7-4c57-8b26-262f7ce01b2b\") " Jul 27 21:05:54.428504 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:54.428391 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1d5d319e-38b7-4c57-8b26-262f7ce01b2b-service-ca\") pod \"1d5d319e-38b7-4c57-8b26-262f7ce01b2b\" (UID: \"1d5d319e-38b7-4c57-8b26-262f7ce01b2b\") " Jul 27 21:05:54.428504 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:54.428423 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zzflr\" (UniqueName: \"kubernetes.io/projected/1d5d319e-38b7-4c57-8b26-262f7ce01b2b-kube-api-access-zzflr\") pod \"1d5d319e-38b7-4c57-8b26-262f7ce01b2b\" (UID: \"1d5d319e-38b7-4c57-8b26-262f7ce01b2b\") " Jul 27 21:05:54.428504 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:54.428448 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d5d319e-38b7-4c57-8b26-262f7ce01b2b-console-serving-cert\") pod \"1d5d319e-38b7-4c57-8b26-262f7ce01b2b\" (UID: \"1d5d319e-38b7-4c57-8b26-262f7ce01b2b\") " Jul 27 21:05:54.428817 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:54.428787 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d5d319e-38b7-4c57-8b26-262f7ce01b2b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1d5d319e-38b7-4c57-8b26-262f7ce01b2b" (UID: "1d5d319e-38b7-4c57-8b26-262f7ce01b2b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 21:05:54.428928 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:54.428907 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d5d319e-38b7-4c57-8b26-262f7ce01b2b-console-config" (OuterVolumeSpecName: "console-config") pod "1d5d319e-38b7-4c57-8b26-262f7ce01b2b" (UID: "1d5d319e-38b7-4c57-8b26-262f7ce01b2b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 21:05:54.428976 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:54.428919 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d5d319e-38b7-4c57-8b26-262f7ce01b2b-service-ca" (OuterVolumeSpecName: "service-ca") pod "1d5d319e-38b7-4c57-8b26-262f7ce01b2b" (UID: "1d5d319e-38b7-4c57-8b26-262f7ce01b2b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 21:05:54.430894 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:54.430864 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d5d319e-38b7-4c57-8b26-262f7ce01b2b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1d5d319e-38b7-4c57-8b26-262f7ce01b2b" (UID: "1d5d319e-38b7-4c57-8b26-262f7ce01b2b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 21:05:54.431006 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:54.430870 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d5d319e-38b7-4c57-8b26-262f7ce01b2b-kube-api-access-zzflr" (OuterVolumeSpecName: "kube-api-access-zzflr") pod "1d5d319e-38b7-4c57-8b26-262f7ce01b2b" (UID: "1d5d319e-38b7-4c57-8b26-262f7ce01b2b"). InnerVolumeSpecName "kube-api-access-zzflr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 21:05:54.431006 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:54.430904 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d5d319e-38b7-4c57-8b26-262f7ce01b2b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1d5d319e-38b7-4c57-8b26-262f7ce01b2b" (UID: "1d5d319e-38b7-4c57-8b26-262f7ce01b2b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 21:05:54.529314 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:54.529265 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzflr\" (UniqueName: \"kubernetes.io/projected/1d5d319e-38b7-4c57-8b26-262f7ce01b2b-kube-api-access-zzflr\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\"" Jul 27 21:05:54.529314 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:54.529303 2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d5d319e-38b7-4c57-8b26-262f7ce01b2b-console-serving-cert\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\"" Jul 27 21:05:54.529314 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:54.529316 2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d5d319e-38b7-4c57-8b26-262f7ce01b2b-oauth-serving-cert\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\"" Jul 27 21:05:54.529563 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:54.529333 2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d5d319e-38b7-4c57-8b26-262f7ce01b2b-console-oauth-config\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\"" Jul 27 21:05:54.529563 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:54.529345 2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1d5d319e-38b7-4c57-8b26-262f7ce01b2b-console-config\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\"" Jul 27 21:05:54.529563 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:54.529357 2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1d5d319e-38b7-4c57-8b26-262f7ce01b2b-service-ca\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\"" Jul 27 21:05:55.081271 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:55.081224 2577 generic.go:356] "Generic (PLEG): container finished" podID="1d5d319e-38b7-4c57-8b26-262f7ce01b2b" containerID="fc4b9ef120820ea7bde15c1eaa51b5a299d931f97997f48840e0cdd13c0764fd" exitCode=0 Jul 27 21:05:55.081787 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:55.081295 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85b49db84c-gv84n" Jul 27 21:05:55.081787 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:55.081309 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85b49db84c-gv84n" event={"ID":"1d5d319e-38b7-4c57-8b26-262f7ce01b2b","Type":"ContainerDied","Data":"fc4b9ef120820ea7bde15c1eaa51b5a299d931f97997f48840e0cdd13c0764fd"} Jul 27 21:05:55.081787 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:55.081406 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85b49db84c-gv84n" event={"ID":"1d5d319e-38b7-4c57-8b26-262f7ce01b2b","Type":"ContainerDied","Data":"6438ae0da76edcfc6c6f8cb04aab5295040fd3f931e7ae72fcce3d226696f13f"} Jul 27 21:05:55.081787 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:55.081441 2577 scope.go:117] "RemoveContainer" containerID="fc4b9ef120820ea7bde15c1eaa51b5a299d931f97997f48840e0cdd13c0764fd" Jul 27 21:05:55.090726 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:55.090699 2577 scope.go:117] "RemoveContainer" containerID="fc4b9ef120820ea7bde15c1eaa51b5a299d931f97997f48840e0cdd13c0764fd" Jul 27 21:05:55.091096 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:05:55.091074 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc4b9ef120820ea7bde15c1eaa51b5a299d931f97997f48840e0cdd13c0764fd\": container with ID starting with fc4b9ef120820ea7bde15c1eaa51b5a299d931f97997f48840e0cdd13c0764fd not found: ID does not exist" containerID="fc4b9ef120820ea7bde15c1eaa51b5a299d931f97997f48840e0cdd13c0764fd" Jul 27 21:05:55.091150 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:55.091106 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc4b9ef120820ea7bde15c1eaa51b5a299d931f97997f48840e0cdd13c0764fd"} err="failed to get container status \"fc4b9ef120820ea7bde15c1eaa51b5a299d931f97997f48840e0cdd13c0764fd\": rpc error: code = NotFound desc = could not find container \"fc4b9ef120820ea7bde15c1eaa51b5a299d931f97997f48840e0cdd13c0764fd\": container with ID starting with fc4b9ef120820ea7bde15c1eaa51b5a299d931f97997f48840e0cdd13c0764fd not found: ID does not exist" Jul 27 21:05:55.106542 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:55.106508 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85b49db84c-gv84n"] Jul 27 21:05:55.112888 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:55.112861 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-85b49db84c-gv84n"] Jul 27 21:05:56.365590 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:05:56.365548 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d5d319e-38b7-4c57-8b26-262f7ce01b2b" path="/var/lib/kubelet/pods/1d5d319e-38b7-4c57-8b26-262f7ce01b2b/volumes" Jul 27 21:06:26.322017 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:26.321934 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-fc8b4fcf-qkjlf"] Jul 27 21:06:26.322457 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:26.322207 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d5d319e-38b7-4c57-8b26-262f7ce01b2b" containerName="console" Jul 27 21:06:26.322457 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:26.322218 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d5d319e-38b7-4c57-8b26-262f7ce01b2b" containerName="console" Jul 27 21:06:26.322457 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:26.322228 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb9ca170-7ea1-45de-aca4-54b4731ed82d" containerName="registry" Jul 27 21:06:26.322457 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:26.322233 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb9ca170-7ea1-45de-aca4-54b4731ed82d" containerName="registry" Jul 27 21:06:26.322457 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:26.322285 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb9ca170-7ea1-45de-aca4-54b4731ed82d" containerName="registry" Jul 27 21:06:26.322457 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:26.322292 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d5d319e-38b7-4c57-8b26-262f7ce01b2b" containerName="console" Jul 27 21:06:26.325488 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:26.325465 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-fc8b4fcf-qkjlf" Jul 27 21:06:26.329118 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:26.329095 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 27 21:06:26.329251 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:26.329096 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 27 21:06:26.334165 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:26.334145 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 27 21:06:26.346558 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:26.346529 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 27 21:06:26.346694 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:26.346590 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-tffkk\"" Jul 27 21:06:26.346694 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:26.346638 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 27 21:06:26.398921 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:26.398891 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 27 21:06:26.448593 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:26.448540 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-fc8b4fcf-qkjlf"] Jul 27 21:06:26.491785 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:26.491751 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/129df604-fd1b-446f-b593-0157d55a5ed8-federate-client-tls\") pod \"telemeter-client-fc8b4fcf-qkjlf\" (UID: \"129df604-fd1b-446f-b593-0157d55a5ed8\") " pod="openshift-monitoring/telemeter-client-fc8b4fcf-qkjlf" Jul 27 21:06:26.491785 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:26.491794 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/129df604-fd1b-446f-b593-0157d55a5ed8-telemeter-client-tls\") pod \"telemeter-client-fc8b4fcf-qkjlf\" (UID: \"129df604-fd1b-446f-b593-0157d55a5ed8\") " pod="openshift-monitoring/telemeter-client-fc8b4fcf-qkjlf" Jul 27 21:06:26.492024 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:26.491847 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/129df604-fd1b-446f-b593-0157d55a5ed8-serving-certs-ca-bundle\") pod \"telemeter-client-fc8b4fcf-qkjlf\" (UID: \"129df604-fd1b-446f-b593-0157d55a5ed8\") " pod="openshift-monitoring/telemeter-client-fc8b4fcf-qkjlf" Jul 27 21:06:26.492076 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:26.492043 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/129df604-fd1b-446f-b593-0157d55a5ed8-telemeter-trusted-ca-bundle\") pod \"telemeter-client-fc8b4fcf-qkjlf\" (UID: \"129df604-fd1b-446f-b593-0157d55a5ed8\") " pod="openshift-monitoring/telemeter-client-fc8b4fcf-qkjlf" Jul 27 21:06:26.492192 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:26.492174 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/129df604-fd1b-446f-b593-0157d55a5ed8-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-fc8b4fcf-qkjlf\" (UID: \"129df604-fd1b-446f-b593-0157d55a5ed8\") " pod="openshift-monitoring/telemeter-client-fc8b4fcf-qkjlf" Jul 27 21:06:26.492256 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:26.492212 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/129df604-fd1b-446f-b593-0157d55a5ed8-secret-telemeter-client\") pod \"telemeter-client-fc8b4fcf-qkjlf\" (UID: \"129df604-fd1b-446f-b593-0157d55a5ed8\") " pod="openshift-monitoring/telemeter-client-fc8b4fcf-qkjlf" Jul 27 21:06:26.492256 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:26.492232 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mqzm\" (UniqueName: \"kubernetes.io/projected/129df604-fd1b-446f-b593-0157d55a5ed8-kube-api-access-6mqzm\") pod \"telemeter-client-fc8b4fcf-qkjlf\" (UID: \"129df604-fd1b-446f-b593-0157d55a5ed8\") " pod="openshift-monitoring/telemeter-client-fc8b4fcf-qkjlf" Jul 27 21:06:26.492330 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:26.492268 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/129df604-fd1b-446f-b593-0157d55a5ed8-metrics-client-ca\") pod \"telemeter-client-fc8b4fcf-qkjlf\" (UID: \"129df604-fd1b-446f-b593-0157d55a5ed8\") " pod="openshift-monitoring/telemeter-client-fc8b4fcf-qkjlf" Jul 27 21:06:26.593784 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:26.593697 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/129df604-fd1b-446f-b593-0157d55a5ed8-federate-client-tls\") pod \"telemeter-client-fc8b4fcf-qkjlf\" (UID: \"129df604-fd1b-446f-b593-0157d55a5ed8\") " pod="openshift-monitoring/telemeter-client-fc8b4fcf-qkjlf" Jul 27 21:06:26.593784 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:26.593738 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/129df604-fd1b-446f-b593-0157d55a5ed8-telemeter-client-tls\") pod \"telemeter-client-fc8b4fcf-qkjlf\" (UID: \"129df604-fd1b-446f-b593-0157d55a5ed8\") " pod="openshift-monitoring/telemeter-client-fc8b4fcf-qkjlf" Jul 27 21:06:26.593784 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:26.593760 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/129df604-fd1b-446f-b593-0157d55a5ed8-serving-certs-ca-bundle\") pod \"telemeter-client-fc8b4fcf-qkjlf\" (UID: \"129df604-fd1b-446f-b593-0157d55a5ed8\") " pod="openshift-monitoring/telemeter-client-fc8b4fcf-qkjlf" Jul 27 21:06:26.594047 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:26.593798 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/129df604-fd1b-446f-b593-0157d55a5ed8-telemeter-trusted-ca-bundle\") pod \"telemeter-client-fc8b4fcf-qkjlf\" (UID: \"129df604-fd1b-446f-b593-0157d55a5ed8\") " pod="openshift-monitoring/telemeter-client-fc8b4fcf-qkjlf" Jul 27 21:06:26.594047 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:26.593853 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/129df604-fd1b-446f-b593-0157d55a5ed8-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-fc8b4fcf-qkjlf\" (UID: \"129df604-fd1b-446f-b593-0157d55a5ed8\") " pod="openshift-monitoring/telemeter-client-fc8b4fcf-qkjlf" Jul 27 21:06:26.594047 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:26.593886 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/129df604-fd1b-446f-b593-0157d55a5ed8-secret-telemeter-client\") pod \"telemeter-client-fc8b4fcf-qkjlf\" (UID: \"129df604-fd1b-446f-b593-0157d55a5ed8\") " pod="openshift-monitoring/telemeter-client-fc8b4fcf-qkjlf" Jul 27 21:06:26.594047 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:26.593908 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mqzm\" (UniqueName: \"kubernetes.io/projected/129df604-fd1b-446f-b593-0157d55a5ed8-kube-api-access-6mqzm\") pod \"telemeter-client-fc8b4fcf-qkjlf\" (UID: \"129df604-fd1b-446f-b593-0157d55a5ed8\") " pod="openshift-monitoring/telemeter-client-fc8b4fcf-qkjlf" Jul 27 21:06:26.594341 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:26.594316 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/129df604-fd1b-446f-b593-0157d55a5ed8-metrics-client-ca\") pod \"telemeter-client-fc8b4fcf-qkjlf\" (UID: \"129df604-fd1b-446f-b593-0157d55a5ed8\") " pod="openshift-monitoring/telemeter-client-fc8b4fcf-qkjlf" Jul 27 21:06:26.594549 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:26.594525 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/129df604-fd1b-446f-b593-0157d55a5ed8-serving-certs-ca-bundle\") pod \"telemeter-client-fc8b4fcf-qkjlf\" (UID: \"129df604-fd1b-446f-b593-0157d55a5ed8\") " pod="openshift-monitoring/telemeter-client-fc8b4fcf-qkjlf" Jul 27 21:06:26.594897 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:26.594870 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/129df604-fd1b-446f-b593-0157d55a5ed8-telemeter-trusted-ca-bundle\") pod \"telemeter-client-fc8b4fcf-qkjlf\" (UID: \"129df604-fd1b-446f-b593-0157d55a5ed8\") " pod="openshift-monitoring/telemeter-client-fc8b4fcf-qkjlf" Jul 27 21:06:26.594992 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:26.594924 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/129df604-fd1b-446f-b593-0157d55a5ed8-metrics-client-ca\") pod \"telemeter-client-fc8b4fcf-qkjlf\" (UID: \"129df604-fd1b-446f-b593-0157d55a5ed8\") " pod="openshift-monitoring/telemeter-client-fc8b4fcf-qkjlf" Jul 27 21:06:26.596687 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:26.596660 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/129df604-fd1b-446f-b593-0157d55a5ed8-secret-telemeter-client\") pod \"telemeter-client-fc8b4fcf-qkjlf\" (UID: \"129df604-fd1b-446f-b593-0157d55a5ed8\") " pod="openshift-monitoring/telemeter-client-fc8b4fcf-qkjlf" Jul 27 21:06:26.596770 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:26.596700 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/129df604-fd1b-446f-b593-0157d55a5ed8-federate-client-tls\") pod \"telemeter-client-fc8b4fcf-qkjlf\" (UID: \"129df604-fd1b-446f-b593-0157d55a5ed8\") " pod="openshift-monitoring/telemeter-client-fc8b4fcf-qkjlf" Jul 27 21:06:26.596770 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:26.596755 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/129df604-fd1b-446f-b593-0157d55a5ed8-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-fc8b4fcf-qkjlf\" (UID: \"129df604-fd1b-446f-b593-0157d55a5ed8\") " pod="openshift-monitoring/telemeter-client-fc8b4fcf-qkjlf" Jul 27 21:06:26.596967 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:26.596947 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/129df604-fd1b-446f-b593-0157d55a5ed8-telemeter-client-tls\") pod \"telemeter-client-fc8b4fcf-qkjlf\" (UID: \"129df604-fd1b-446f-b593-0157d55a5ed8\") " pod="openshift-monitoring/telemeter-client-fc8b4fcf-qkjlf" Jul 27 21:06:26.605039 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:26.605014 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mqzm\" (UniqueName: \"kubernetes.io/projected/129df604-fd1b-446f-b593-0157d55a5ed8-kube-api-access-6mqzm\") pod \"telemeter-client-fc8b4fcf-qkjlf\" (UID: \"129df604-fd1b-446f-b593-0157d55a5ed8\") " pod="openshift-monitoring/telemeter-client-fc8b4fcf-qkjlf" Jul 27 21:06:26.634963 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:26.634904 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-fc8b4fcf-qkjlf" Jul 27 21:06:26.798633 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:26.798608 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-fc8b4fcf-qkjlf"] Jul 27 21:06:26.800516 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:06:26.800490 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod129df604_fd1b_446f_b593_0157d55a5ed8.slice/crio-a96513767cb966e78c16ef88110cea2e329abea3f5205dec2fe138764301e4fe WatchSource:0}: Error finding container a96513767cb966e78c16ef88110cea2e329abea3f5205dec2fe138764301e4fe: Status 404 returned error can't find the container with id a96513767cb966e78c16ef88110cea2e329abea3f5205dec2fe138764301e4fe Jul 27 21:06:27.210648 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:27.210606 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-fc8b4fcf-qkjlf" event={"ID":"129df604-fd1b-446f-b593-0157d55a5ed8","Type":"ContainerStarted","Data":"a96513767cb966e78c16ef88110cea2e329abea3f5205dec2fe138764301e4fe"} Jul 27 21:06:29.221163 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:29.221124 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-fc8b4fcf-qkjlf" event={"ID":"129df604-fd1b-446f-b593-0157d55a5ed8","Type":"ContainerStarted","Data":"c1f74e3bf5bcd63941da7225aeed96f660e8a9546a3d2ed5bf90fc050bdbeac0"} Jul 27 21:06:30.227717 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:30.227679 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-fc8b4fcf-qkjlf" event={"ID":"129df604-fd1b-446f-b593-0157d55a5ed8","Type":"ContainerStarted","Data":"59d652b8e36cd7301b0db504bc60bdfdd75e991d1604ad7d01bdcd327043b9b9"} Jul 27 21:06:30.228144 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:30.227767 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-fc8b4fcf-qkjlf" event={"ID":"129df604-fd1b-446f-b593-0157d55a5ed8","Type":"ContainerStarted","Data":"37ff19b7ae725ec651a9fb860460a407a54e66f51a15c99266c6e663769a30ac"} Jul 27 21:06:30.250781 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:30.250732 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-fc8b4fcf-qkjlf" podStartSLOduration=1.696265558 podStartE2EDuration="4.250693791s" totalImagesPullingTime="2.554428233s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:06:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:06:30.249958857 +0000 UTC m=+182.381575081" watchObservedRunningTime="2026-07-27 21:06:30.250693791 +0000 UTC m=+182.382309993" Jul 27 21:06:30.947669 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:30.947631 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5c86bc6544-6qfcp"] Jul 27 21:06:30.951097 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:30.951072 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c86bc6544-6qfcp" Jul 27 21:06:30.963636 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:30.963603 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c86bc6544-6qfcp"] Jul 27 21:06:31.131861 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:31.131809 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6kzt\" (UniqueName: \"kubernetes.io/projected/48cffbd7-f0e4-4308-b8fb-15fed01a4f08-kube-api-access-t6kzt\") pod \"console-5c86bc6544-6qfcp\" (UID: \"48cffbd7-f0e4-4308-b8fb-15fed01a4f08\") " pod="openshift-console/console-5c86bc6544-6qfcp" Jul 27 21:06:31.131861 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:31.131867 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48cffbd7-f0e4-4308-b8fb-15fed01a4f08-trusted-ca-bundle\") pod \"console-5c86bc6544-6qfcp\" (UID: \"48cffbd7-f0e4-4308-b8fb-15fed01a4f08\") " pod="openshift-console/console-5c86bc6544-6qfcp" Jul 27 21:06:31.132111 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:31.131933 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/48cffbd7-f0e4-4308-b8fb-15fed01a4f08-service-ca\") pod \"console-5c86bc6544-6qfcp\" (UID: \"48cffbd7-f0e4-4308-b8fb-15fed01a4f08\") " pod="openshift-console/console-5c86bc6544-6qfcp" Jul 27 21:06:31.132111 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:31.131980 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/48cffbd7-f0e4-4308-b8fb-15fed01a4f08-console-serving-cert\") pod \"console-5c86bc6544-6qfcp\" (UID: \"48cffbd7-f0e4-4308-b8fb-15fed01a4f08\") " pod="openshift-console/console-5c86bc6544-6qfcp" Jul 27 21:06:31.132111 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:31.132003 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/48cffbd7-f0e4-4308-b8fb-15fed01a4f08-oauth-serving-cert\") pod \"console-5c86bc6544-6qfcp\" (UID: \"48cffbd7-f0e4-4308-b8fb-15fed01a4f08\") " pod="openshift-console/console-5c86bc6544-6qfcp" Jul 27 21:06:31.132259 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:31.132126 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/48cffbd7-f0e4-4308-b8fb-15fed01a4f08-console-oauth-config\") pod \"console-5c86bc6544-6qfcp\" (UID: \"48cffbd7-f0e4-4308-b8fb-15fed01a4f08\") " pod="openshift-console/console-5c86bc6544-6qfcp" Jul 27 21:06:31.132259 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:31.132161 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/48cffbd7-f0e4-4308-b8fb-15fed01a4f08-console-config\") pod \"console-5c86bc6544-6qfcp\" (UID: \"48cffbd7-f0e4-4308-b8fb-15fed01a4f08\") " pod="openshift-console/console-5c86bc6544-6qfcp" Jul 27 21:06:31.233229 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:31.233196 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/48cffbd7-f0e4-4308-b8fb-15fed01a4f08-console-oauth-config\") pod \"console-5c86bc6544-6qfcp\" (UID: \"48cffbd7-f0e4-4308-b8fb-15fed01a4f08\") " pod="openshift-console/console-5c86bc6544-6qfcp" Jul 27 21:06:31.233229 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:31.233234 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/48cffbd7-f0e4-4308-b8fb-15fed01a4f08-console-config\") pod \"console-5c86bc6544-6qfcp\" (UID: \"48cffbd7-f0e4-4308-b8fb-15fed01a4f08\") " pod="openshift-console/console-5c86bc6544-6qfcp" Jul 27 21:06:31.233820 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:31.233256 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6kzt\" (UniqueName: \"kubernetes.io/projected/48cffbd7-f0e4-4308-b8fb-15fed01a4f08-kube-api-access-t6kzt\") pod \"console-5c86bc6544-6qfcp\" (UID: \"48cffbd7-f0e4-4308-b8fb-15fed01a4f08\") " pod="openshift-console/console-5c86bc6544-6qfcp" Jul 27 21:06:31.233820 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:31.233279 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48cffbd7-f0e4-4308-b8fb-15fed01a4f08-trusted-ca-bundle\") pod \"console-5c86bc6544-6qfcp\" (UID: \"48cffbd7-f0e4-4308-b8fb-15fed01a4f08\") " pod="openshift-console/console-5c86bc6544-6qfcp" Jul 27 21:06:31.233820 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:31.233303 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/48cffbd7-f0e4-4308-b8fb-15fed01a4f08-service-ca\") pod \"console-5c86bc6544-6qfcp\" (UID: \"48cffbd7-f0e4-4308-b8fb-15fed01a4f08\") " pod="openshift-console/console-5c86bc6544-6qfcp" Jul 27 21:06:31.233820 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:31.233323 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/48cffbd7-f0e4-4308-b8fb-15fed01a4f08-console-serving-cert\") pod \"console-5c86bc6544-6qfcp\" (UID: \"48cffbd7-f0e4-4308-b8fb-15fed01a4f08\") " pod="openshift-console/console-5c86bc6544-6qfcp" Jul 27 21:06:31.233820 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:31.233344 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/48cffbd7-f0e4-4308-b8fb-15fed01a4f08-oauth-serving-cert\") pod \"console-5c86bc6544-6qfcp\" (UID: \"48cffbd7-f0e4-4308-b8fb-15fed01a4f08\") " pod="openshift-console/console-5c86bc6544-6qfcp" Jul 27 21:06:31.234187 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:31.234162 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/48cffbd7-f0e4-4308-b8fb-15fed01a4f08-console-config\") pod \"console-5c86bc6544-6qfcp\" (UID: \"48cffbd7-f0e4-4308-b8fb-15fed01a4f08\") " pod="openshift-console/console-5c86bc6544-6qfcp" Jul 27 21:06:31.234261 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:31.234168 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/48cffbd7-f0e4-4308-b8fb-15fed01a4f08-oauth-serving-cert\") pod \"console-5c86bc6544-6qfcp\" (UID: \"48cffbd7-f0e4-4308-b8fb-15fed01a4f08\") " pod="openshift-console/console-5c86bc6544-6qfcp" Jul 27 21:06:31.234261 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:31.234168 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/48cffbd7-f0e4-4308-b8fb-15fed01a4f08-service-ca\") pod \"console-5c86bc6544-6qfcp\" (UID: \"48cffbd7-f0e4-4308-b8fb-15fed01a4f08\") " pod="openshift-console/console-5c86bc6544-6qfcp" Jul 27 21:06:31.234423 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:31.234399 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48cffbd7-f0e4-4308-b8fb-15fed01a4f08-trusted-ca-bundle\") pod \"console-5c86bc6544-6qfcp\" (UID: \"48cffbd7-f0e4-4308-b8fb-15fed01a4f08\") " pod="openshift-console/console-5c86bc6544-6qfcp" Jul 27 21:06:31.235979 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:31.235957 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/48cffbd7-f0e4-4308-b8fb-15fed01a4f08-console-serving-cert\") pod \"console-5c86bc6544-6qfcp\" (UID: \"48cffbd7-f0e4-4308-b8fb-15fed01a4f08\") " pod="openshift-console/console-5c86bc6544-6qfcp" Jul 27 21:06:31.235979 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:31.235971 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/48cffbd7-f0e4-4308-b8fb-15fed01a4f08-console-oauth-config\") pod \"console-5c86bc6544-6qfcp\" (UID: \"48cffbd7-f0e4-4308-b8fb-15fed01a4f08\") " pod="openshift-console/console-5c86bc6544-6qfcp" Jul 27 21:06:31.242439 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:31.242410 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6kzt\" (UniqueName: \"kubernetes.io/projected/48cffbd7-f0e4-4308-b8fb-15fed01a4f08-kube-api-access-t6kzt\") pod \"console-5c86bc6544-6qfcp\" (UID: \"48cffbd7-f0e4-4308-b8fb-15fed01a4f08\") " pod="openshift-console/console-5c86bc6544-6qfcp" Jul 27 21:06:31.262077 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:31.262041 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c86bc6544-6qfcp" Jul 27 21:06:31.412768 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:31.412728 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c86bc6544-6qfcp"] Jul 27 21:06:31.415615 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:06:31.415560 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48cffbd7_f0e4_4308_b8fb_15fed01a4f08.slice/crio-6bb3828ee354fb2bd46bdc2acc2b6ccb0ec670dafdb3252a02ad2503b193bea2 WatchSource:0}: Error finding container 6bb3828ee354fb2bd46bdc2acc2b6ccb0ec670dafdb3252a02ad2503b193bea2: Status 404 returned error can't find the container with id 6bb3828ee354fb2bd46bdc2acc2b6ccb0ec670dafdb3252a02ad2503b193bea2 Jul 27 21:06:32.237547 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:32.237506 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c86bc6544-6qfcp" event={"ID":"48cffbd7-f0e4-4308-b8fb-15fed01a4f08","Type":"ContainerStarted","Data":"7dcabc13eb389647855a63163f7a58955999a55c1680fcb73437b1dbe5f481d8"} Jul 27 21:06:32.238018 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:32.237582 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c86bc6544-6qfcp" event={"ID":"48cffbd7-f0e4-4308-b8fb-15fed01a4f08","Type":"ContainerStarted","Data":"6bb3828ee354fb2bd46bdc2acc2b6ccb0ec670dafdb3252a02ad2503b193bea2"} Jul 27 21:06:32.257619 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:32.257549 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5c86bc6544-6qfcp" podStartSLOduration=2.2575291760000002 podStartE2EDuration="2.257529176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:06:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:06:32.255492857 +0000 UTC m=+184.387109082" watchObservedRunningTime="2026-07-27 21:06:32.257529176 +0000 UTC m=+184.389145464" Jul 27 21:06:41.262747 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:41.262700 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5c86bc6544-6qfcp" Jul 27 21:06:41.263205 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:41.262778 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5c86bc6544-6qfcp" Jul 27 21:06:41.267596 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:41.267552 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5c86bc6544-6qfcp" Jul 27 21:06:41.275827 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:41.275804 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5c86bc6544-6qfcp" Jul 27 21:06:41.329779 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:06:41.329746 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-776bb4b4b8-djl4p"] Jul 27 21:07:06.352316 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:06.352267 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-776bb4b4b8-djl4p" podUID="0b07e3ce-07f7-4df9-8d82-d58cf7a3b285" containerName="console" containerID="cri-o://64151aadf9c9f46cdba8a9fca35a7ac77ecec43c28ef9de8ef44171047b1a4e8" gracePeriod=15 Jul 27 21:07:06.595829 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:06.595802 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-776bb4b4b8-djl4p" Jul 27 21:07:06.724440 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:06.724407 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0b07e3ce-07f7-4df9-8d82-d58cf7a3b285-service-ca\") pod \"0b07e3ce-07f7-4df9-8d82-d58cf7a3b285\" (UID: \"0b07e3ce-07f7-4df9-8d82-d58cf7a3b285\") " Jul 27 21:07:06.724627 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:06.724448 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0b07e3ce-07f7-4df9-8d82-d58cf7a3b285-console-oauth-config\") pod \"0b07e3ce-07f7-4df9-8d82-d58cf7a3b285\" (UID: \"0b07e3ce-07f7-4df9-8d82-d58cf7a3b285\") " Jul 27 21:07:06.724627 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:06.724473 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0b07e3ce-07f7-4df9-8d82-d58cf7a3b285-console-config\") pod \"0b07e3ce-07f7-4df9-8d82-d58cf7a3b285\" (UID: \"0b07e3ce-07f7-4df9-8d82-d58cf7a3b285\") " Jul 27 21:07:06.724627 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:06.724561 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0b07e3ce-07f7-4df9-8d82-d58cf7a3b285-console-serving-cert\") pod \"0b07e3ce-07f7-4df9-8d82-d58cf7a3b285\" (UID: \"0b07e3ce-07f7-4df9-8d82-d58cf7a3b285\") " Jul 27 21:07:06.724733 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:06.724635 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b07e3ce-07f7-4df9-8d82-d58cf7a3b285-trusted-ca-bundle\") pod \"0b07e3ce-07f7-4df9-8d82-d58cf7a3b285\" (UID: \"0b07e3ce-07f7-4df9-8d82-d58cf7a3b285\") " Jul 27 21:07:06.724733 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:06.724705 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0b07e3ce-07f7-4df9-8d82-d58cf7a3b285-oauth-serving-cert\") pod \"0b07e3ce-07f7-4df9-8d82-d58cf7a3b285\" (UID: \"0b07e3ce-07f7-4df9-8d82-d58cf7a3b285\") " Jul 27 21:07:06.724827 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:06.724753 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hm4pd\" (UniqueName: \"kubernetes.io/projected/0b07e3ce-07f7-4df9-8d82-d58cf7a3b285-kube-api-access-hm4pd\") pod \"0b07e3ce-07f7-4df9-8d82-d58cf7a3b285\" (UID: \"0b07e3ce-07f7-4df9-8d82-d58cf7a3b285\") " Jul 27 21:07:06.724907 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:06.724886 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b07e3ce-07f7-4df9-8d82-d58cf7a3b285-service-ca" (OuterVolumeSpecName: "service-ca") pod "0b07e3ce-07f7-4df9-8d82-d58cf7a3b285" (UID: "0b07e3ce-07f7-4df9-8d82-d58cf7a3b285"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 21:07:06.724970 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:06.724940 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b07e3ce-07f7-4df9-8d82-d58cf7a3b285-console-config" (OuterVolumeSpecName: "console-config") pod "0b07e3ce-07f7-4df9-8d82-d58cf7a3b285" (UID: "0b07e3ce-07f7-4df9-8d82-d58cf7a3b285"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 21:07:06.725026 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:06.724996 2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0b07e3ce-07f7-4df9-8d82-d58cf7a3b285-service-ca\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\"" Jul 27 21:07:06.725171 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:06.725134 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b07e3ce-07f7-4df9-8d82-d58cf7a3b285-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "0b07e3ce-07f7-4df9-8d82-d58cf7a3b285" (UID: "0b07e3ce-07f7-4df9-8d82-d58cf7a3b285"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 21:07:06.725171 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:06.725146 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b07e3ce-07f7-4df9-8d82-d58cf7a3b285-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "0b07e3ce-07f7-4df9-8d82-d58cf7a3b285" (UID: "0b07e3ce-07f7-4df9-8d82-d58cf7a3b285"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 21:07:06.726760 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:06.726732 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b07e3ce-07f7-4df9-8d82-d58cf7a3b285-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "0b07e3ce-07f7-4df9-8d82-d58cf7a3b285" (UID: "0b07e3ce-07f7-4df9-8d82-d58cf7a3b285"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 21:07:06.726760 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:06.726752 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b07e3ce-07f7-4df9-8d82-d58cf7a3b285-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "0b07e3ce-07f7-4df9-8d82-d58cf7a3b285" (UID: "0b07e3ce-07f7-4df9-8d82-d58cf7a3b285"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 21:07:06.726890 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:06.726774 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b07e3ce-07f7-4df9-8d82-d58cf7a3b285-kube-api-access-hm4pd" (OuterVolumeSpecName: "kube-api-access-hm4pd") pod "0b07e3ce-07f7-4df9-8d82-d58cf7a3b285" (UID: "0b07e3ce-07f7-4df9-8d82-d58cf7a3b285"). InnerVolumeSpecName "kube-api-access-hm4pd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 21:07:06.825517 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:06.825470 2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0b07e3ce-07f7-4df9-8d82-d58cf7a3b285-console-config\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\"" Jul 27 21:07:06.825517 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:06.825509 2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0b07e3ce-07f7-4df9-8d82-d58cf7a3b285-console-serving-cert\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\"" Jul 27 21:07:06.825517 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:06.825522 2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b07e3ce-07f7-4df9-8d82-d58cf7a3b285-trusted-ca-bundle\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\"" Jul 27 21:07:06.825791 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:06.825535 2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0b07e3ce-07f7-4df9-8d82-d58cf7a3b285-oauth-serving-cert\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\"" Jul 27 21:07:06.825791 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:06.825548 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hm4pd\" (UniqueName: \"kubernetes.io/projected/0b07e3ce-07f7-4df9-8d82-d58cf7a3b285-kube-api-access-hm4pd\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\"" Jul 27 21:07:06.825791 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:06.825561 2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0b07e3ce-07f7-4df9-8d82-d58cf7a3b285-console-oauth-config\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\"" Jul 27 21:07:07.370919 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:07.370881 2577 generic.go:356] "Generic (PLEG): container finished" podID="0b07e3ce-07f7-4df9-8d82-d58cf7a3b285" containerID="64151aadf9c9f46cdba8a9fca35a7ac77ecec43c28ef9de8ef44171047b1a4e8" exitCode=0 Jul 27 21:07:07.371358 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:07.370938 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-776bb4b4b8-djl4p" Jul 27 21:07:07.371358 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:07.370959 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-776bb4b4b8-djl4p" event={"ID":"0b07e3ce-07f7-4df9-8d82-d58cf7a3b285","Type":"ContainerDied","Data":"64151aadf9c9f46cdba8a9fca35a7ac77ecec43c28ef9de8ef44171047b1a4e8"} Jul 27 21:07:07.371358 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:07.371009 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-776bb4b4b8-djl4p" event={"ID":"0b07e3ce-07f7-4df9-8d82-d58cf7a3b285","Type":"ContainerDied","Data":"a37ef76959eeb996a28cf79943a0f73bc27d0b507063fdd89432c672cf7d3c05"} Jul 27 21:07:07.371358 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:07.371029 2577 scope.go:117] "RemoveContainer" containerID="64151aadf9c9f46cdba8a9fca35a7ac77ecec43c28ef9de8ef44171047b1a4e8" Jul 27 21:07:07.381756 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:07.381736 2577 scope.go:117] "RemoveContainer" containerID="64151aadf9c9f46cdba8a9fca35a7ac77ecec43c28ef9de8ef44171047b1a4e8" Jul 27 21:07:07.382050 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:07:07.382029 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64151aadf9c9f46cdba8a9fca35a7ac77ecec43c28ef9de8ef44171047b1a4e8\": container with ID starting with 64151aadf9c9f46cdba8a9fca35a7ac77ecec43c28ef9de8ef44171047b1a4e8 not found: ID does not exist" containerID="64151aadf9c9f46cdba8a9fca35a7ac77ecec43c28ef9de8ef44171047b1a4e8" Jul 27 21:07:07.382122 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:07.382063 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64151aadf9c9f46cdba8a9fca35a7ac77ecec43c28ef9de8ef44171047b1a4e8"} err="failed to get container status \"64151aadf9c9f46cdba8a9fca35a7ac77ecec43c28ef9de8ef44171047b1a4e8\": rpc error: code = NotFound desc = could not find container \"64151aadf9c9f46cdba8a9fca35a7ac77ecec43c28ef9de8ef44171047b1a4e8\": container with ID starting with 64151aadf9c9f46cdba8a9fca35a7ac77ecec43c28ef9de8ef44171047b1a4e8 not found: ID does not exist" Jul 27 21:07:07.392027 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:07.391996 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-776bb4b4b8-djl4p"] Jul 27 21:07:07.398982 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:07.398951 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-776bb4b4b8-djl4p"] Jul 27 21:07:08.365015 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:08.364979 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b07e3ce-07f7-4df9-8d82-d58cf7a3b285" path="/var/lib/kubelet/pods/0b07e3ce-07f7-4df9-8d82-d58cf7a3b285/volumes" Jul 27 21:07:46.662362 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:46.662326 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-dcbd5694-ntknm"] Jul 27 21:07:46.662876 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:46.662654 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b07e3ce-07f7-4df9-8d82-d58cf7a3b285" containerName="console" Jul 27 21:07:46.662876 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:46.662669 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b07e3ce-07f7-4df9-8d82-d58cf7a3b285" containerName="console" Jul 27 21:07:46.662876 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:46.662734 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b07e3ce-07f7-4df9-8d82-d58cf7a3b285" containerName="console" Jul 27 21:07:46.667045 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:46.667019 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dcbd5694-ntknm" Jul 27 21:07:46.679688 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:46.679659 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-dcbd5694-ntknm"] Jul 27 21:07:46.745257 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:46.745216 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef24c526-4136-46ba-973e-b570b043aff6-trusted-ca-bundle\") pod \"console-dcbd5694-ntknm\" (UID: \"ef24c526-4136-46ba-973e-b570b043aff6\") " pod="openshift-console/console-dcbd5694-ntknm" Jul 27 21:07:46.745432 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:46.745275 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ef24c526-4136-46ba-973e-b570b043aff6-console-oauth-config\") pod \"console-dcbd5694-ntknm\" (UID: \"ef24c526-4136-46ba-973e-b570b043aff6\") " pod="openshift-console/console-dcbd5694-ntknm" Jul 27 21:07:46.745432 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:46.745347 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ef24c526-4136-46ba-973e-b570b043aff6-service-ca\") pod \"console-dcbd5694-ntknm\" (UID: \"ef24c526-4136-46ba-973e-b570b043aff6\") " pod="openshift-console/console-dcbd5694-ntknm" Jul 27 21:07:46.745432 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:46.745384 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbmb6\" (UniqueName: \"kubernetes.io/projected/ef24c526-4136-46ba-973e-b570b043aff6-kube-api-access-dbmb6\") pod \"console-dcbd5694-ntknm\" (UID: \"ef24c526-4136-46ba-973e-b570b043aff6\") " pod="openshift-console/console-dcbd5694-ntknm" Jul 27 21:07:46.745432 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:46.745410 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ef24c526-4136-46ba-973e-b570b043aff6-oauth-serving-cert\") pod \"console-dcbd5694-ntknm\" (UID: \"ef24c526-4136-46ba-973e-b570b043aff6\") " pod="openshift-console/console-dcbd5694-ntknm" Jul 27 21:07:46.745557 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:46.745470 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ef24c526-4136-46ba-973e-b570b043aff6-console-serving-cert\") pod \"console-dcbd5694-ntknm\" (UID: \"ef24c526-4136-46ba-973e-b570b043aff6\") " pod="openshift-console/console-dcbd5694-ntknm" Jul 27 21:07:46.745557 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:46.745507 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ef24c526-4136-46ba-973e-b570b043aff6-console-config\") pod \"console-dcbd5694-ntknm\" (UID: \"ef24c526-4136-46ba-973e-b570b043aff6\") " pod="openshift-console/console-dcbd5694-ntknm" Jul 27 21:07:46.846309 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:46.846268 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ef24c526-4136-46ba-973e-b570b043aff6-console-oauth-config\") pod \"console-dcbd5694-ntknm\" (UID: \"ef24c526-4136-46ba-973e-b570b043aff6\") " pod="openshift-console/console-dcbd5694-ntknm" Jul 27 21:07:46.846494 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:46.846320 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ef24c526-4136-46ba-973e-b570b043aff6-service-ca\") pod \"console-dcbd5694-ntknm\" (UID: \"ef24c526-4136-46ba-973e-b570b043aff6\") " pod="openshift-console/console-dcbd5694-ntknm" Jul 27 21:07:46.846494 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:46.846343 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbmb6\" (UniqueName: \"kubernetes.io/projected/ef24c526-4136-46ba-973e-b570b043aff6-kube-api-access-dbmb6\") pod \"console-dcbd5694-ntknm\" (UID: \"ef24c526-4136-46ba-973e-b570b043aff6\") " pod="openshift-console/console-dcbd5694-ntknm" Jul 27 21:07:46.846494 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:46.846361 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ef24c526-4136-46ba-973e-b570b043aff6-oauth-serving-cert\") pod \"console-dcbd5694-ntknm\" (UID: \"ef24c526-4136-46ba-973e-b570b043aff6\") " pod="openshift-console/console-dcbd5694-ntknm" Jul 27 21:07:46.846494 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:46.846381 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ef24c526-4136-46ba-973e-b570b043aff6-console-serving-cert\") pod \"console-dcbd5694-ntknm\" (UID: \"ef24c526-4136-46ba-973e-b570b043aff6\") " pod="openshift-console/console-dcbd5694-ntknm" Jul 27 21:07:46.846494 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:46.846399 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ef24c526-4136-46ba-973e-b570b043aff6-console-config\") pod \"console-dcbd5694-ntknm\" (UID: \"ef24c526-4136-46ba-973e-b570b043aff6\") " pod="openshift-console/console-dcbd5694-ntknm" Jul 27 21:07:46.846494 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:46.846427 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef24c526-4136-46ba-973e-b570b043aff6-trusted-ca-bundle\") pod \"console-dcbd5694-ntknm\" (UID: \"ef24c526-4136-46ba-973e-b570b043aff6\") " pod="openshift-console/console-dcbd5694-ntknm" Jul 27 21:07:46.847243 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:46.847210 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ef24c526-4136-46ba-973e-b570b043aff6-oauth-serving-cert\") pod \"console-dcbd5694-ntknm\" (UID: \"ef24c526-4136-46ba-973e-b570b043aff6\") " pod="openshift-console/console-dcbd5694-ntknm" Jul 27 21:07:46.847368 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:46.847216 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ef24c526-4136-46ba-973e-b570b043aff6-service-ca\") pod \"console-dcbd5694-ntknm\" (UID: \"ef24c526-4136-46ba-973e-b570b043aff6\") " pod="openshift-console/console-dcbd5694-ntknm" Jul 27 21:07:46.847368 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:46.847303 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ef24c526-4136-46ba-973e-b570b043aff6-console-config\") pod \"console-dcbd5694-ntknm\" (UID: \"ef24c526-4136-46ba-973e-b570b043aff6\") " pod="openshift-console/console-dcbd5694-ntknm" Jul 27 21:07:46.847506 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:46.847483 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef24c526-4136-46ba-973e-b570b043aff6-trusted-ca-bundle\") pod \"console-dcbd5694-ntknm\" (UID: \"ef24c526-4136-46ba-973e-b570b043aff6\") " pod="openshift-console/console-dcbd5694-ntknm" Jul 27 21:07:46.848934 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:46.848910 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ef24c526-4136-46ba-973e-b570b043aff6-console-oauth-config\") pod \"console-dcbd5694-ntknm\" (UID: \"ef24c526-4136-46ba-973e-b570b043aff6\") " pod="openshift-console/console-dcbd5694-ntknm" Jul 27 21:07:46.849037 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:46.849027 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ef24c526-4136-46ba-973e-b570b043aff6-console-serving-cert\") pod \"console-dcbd5694-ntknm\" (UID: \"ef24c526-4136-46ba-973e-b570b043aff6\") " pod="openshift-console/console-dcbd5694-ntknm" Jul 27 21:07:46.855303 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:46.855275 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbmb6\" (UniqueName: \"kubernetes.io/projected/ef24c526-4136-46ba-973e-b570b043aff6-kube-api-access-dbmb6\") pod \"console-dcbd5694-ntknm\" (UID: \"ef24c526-4136-46ba-973e-b570b043aff6\") " pod="openshift-console/console-dcbd5694-ntknm" Jul 27 21:07:46.976670 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:46.976633 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dcbd5694-ntknm" Jul 27 21:07:47.112974 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:47.112946 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-dcbd5694-ntknm"] Jul 27 21:07:47.115598 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:07:47.115554 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef24c526_4136_46ba_973e_b570b043aff6.slice/crio-7aa0d8e83e2d65dc7985e5c4d72390955b0d7165f771709f264f0fbff350e02f WatchSource:0}: Error finding container 7aa0d8e83e2d65dc7985e5c4d72390955b0d7165f771709f264f0fbff350e02f: Status 404 returned error can't find the container with id 7aa0d8e83e2d65dc7985e5c4d72390955b0d7165f771709f264f0fbff350e02f Jul 27 21:07:47.522083 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:47.522040 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dcbd5694-ntknm" event={"ID":"ef24c526-4136-46ba-973e-b570b043aff6","Type":"ContainerStarted","Data":"1c0cac8c8e93c336b9ce4b9f35ba0531364a4a4da6679b29e397f486330cf060"} Jul 27 21:07:47.522268 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:47.522122 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dcbd5694-ntknm" event={"ID":"ef24c526-4136-46ba-973e-b570b043aff6","Type":"ContainerStarted","Data":"7aa0d8e83e2d65dc7985e5c4d72390955b0d7165f771709f264f0fbff350e02f"} Jul 27 21:07:47.543092 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:47.543038 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-dcbd5694-ntknm" podStartSLOduration=1.543016376 podStartE2EDuration="1.543016376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:07:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:07:47.542316489 +0000 UTC m=+259.673932713" watchObservedRunningTime="2026-07-27 21:07:47.543016376 +0000 UTC m=+259.674632599" Jul 27 21:07:56.976838 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:56.976742 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-dcbd5694-ntknm" Jul 27 21:07:56.977232 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:56.976903 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-dcbd5694-ntknm" Jul 27 21:07:56.981684 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:56.981656 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-dcbd5694-ntknm" Jul 27 21:07:57.562471 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:57.562438 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-dcbd5694-ntknm" Jul 27 21:07:57.611019 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:07:57.610983 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c86bc6544-6qfcp"] Jul 27 21:08:22.632814 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:08:22.632748 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5c86bc6544-6qfcp" podUID="48cffbd7-f0e4-4308-b8fb-15fed01a4f08" containerName="console" containerID="cri-o://7dcabc13eb389647855a63163f7a58955999a55c1680fcb73437b1dbe5f481d8" gracePeriod=15 Jul 27 21:08:22.873355 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:08:22.873329 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c86bc6544-6qfcp" Jul 27 21:08:23.045112 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:08:23.045077 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/48cffbd7-f0e4-4308-b8fb-15fed01a4f08-console-oauth-config\") pod \"48cffbd7-f0e4-4308-b8fb-15fed01a4f08\" (UID: \"48cffbd7-f0e4-4308-b8fb-15fed01a4f08\") " Jul 27 21:08:23.045304 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:08:23.045129 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/48cffbd7-f0e4-4308-b8fb-15fed01a4f08-oauth-serving-cert\") pod \"48cffbd7-f0e4-4308-b8fb-15fed01a4f08\" (UID: \"48cffbd7-f0e4-4308-b8fb-15fed01a4f08\") " Jul 27 21:08:23.045304 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:08:23.045163 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/48cffbd7-f0e4-4308-b8fb-15fed01a4f08-console-serving-cert\") pod \"48cffbd7-f0e4-4308-b8fb-15fed01a4f08\" (UID: \"48cffbd7-f0e4-4308-b8fb-15fed01a4f08\") " Jul 27 21:08:23.045304 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:08:23.045200 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48cffbd7-f0e4-4308-b8fb-15fed01a4f08-trusted-ca-bundle\") pod \"48cffbd7-f0e4-4308-b8fb-15fed01a4f08\" (UID: \"48cffbd7-f0e4-4308-b8fb-15fed01a4f08\") " Jul 27 21:08:23.045304 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:08:23.045227 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t6kzt\" (UniqueName: \"kubernetes.io/projected/48cffbd7-f0e4-4308-b8fb-15fed01a4f08-kube-api-access-t6kzt\") pod \"48cffbd7-f0e4-4308-b8fb-15fed01a4f08\" (UID: \"48cffbd7-f0e4-4308-b8fb-15fed01a4f08\") " Jul 27 21:08:23.045304 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:08:23.045250 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/48cffbd7-f0e4-4308-b8fb-15fed01a4f08-console-config\") pod \"48cffbd7-f0e4-4308-b8fb-15fed01a4f08\" (UID: \"48cffbd7-f0e4-4308-b8fb-15fed01a4f08\") " Jul 27 21:08:23.045304 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:08:23.045286 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/48cffbd7-f0e4-4308-b8fb-15fed01a4f08-service-ca\") pod \"48cffbd7-f0e4-4308-b8fb-15fed01a4f08\" (UID: \"48cffbd7-f0e4-4308-b8fb-15fed01a4f08\") " Jul 27 21:08:23.045765 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:08:23.045735 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48cffbd7-f0e4-4308-b8fb-15fed01a4f08-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "48cffbd7-f0e4-4308-b8fb-15fed01a4f08" (UID: "48cffbd7-f0e4-4308-b8fb-15fed01a4f08"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 21:08:23.045848 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:08:23.045763 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48cffbd7-f0e4-4308-b8fb-15fed01a4f08-console-config" (OuterVolumeSpecName: "console-config") pod "48cffbd7-f0e4-4308-b8fb-15fed01a4f08" (UID: "48cffbd7-f0e4-4308-b8fb-15fed01a4f08"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 21:08:23.045848 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:08:23.045776 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48cffbd7-f0e4-4308-b8fb-15fed01a4f08-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "48cffbd7-f0e4-4308-b8fb-15fed01a4f08" (UID: "48cffbd7-f0e4-4308-b8fb-15fed01a4f08"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 21:08:23.045955 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:08:23.045853 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48cffbd7-f0e4-4308-b8fb-15fed01a4f08-service-ca" (OuterVolumeSpecName: "service-ca") pod "48cffbd7-f0e4-4308-b8fb-15fed01a4f08" (UID: "48cffbd7-f0e4-4308-b8fb-15fed01a4f08"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 21:08:23.047362 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:08:23.047332 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48cffbd7-f0e4-4308-b8fb-15fed01a4f08-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "48cffbd7-f0e4-4308-b8fb-15fed01a4f08" (UID: "48cffbd7-f0e4-4308-b8fb-15fed01a4f08"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 21:08:23.047362 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:08:23.047350 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48cffbd7-f0e4-4308-b8fb-15fed01a4f08-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "48cffbd7-f0e4-4308-b8fb-15fed01a4f08" (UID: "48cffbd7-f0e4-4308-b8fb-15fed01a4f08"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 21:08:23.047492 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:08:23.047410 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48cffbd7-f0e4-4308-b8fb-15fed01a4f08-kube-api-access-t6kzt" (OuterVolumeSpecName: "kube-api-access-t6kzt") pod "48cffbd7-f0e4-4308-b8fb-15fed01a4f08" (UID: "48cffbd7-f0e4-4308-b8fb-15fed01a4f08"). InnerVolumeSpecName "kube-api-access-t6kzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 21:08:23.145899 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:08:23.145860 2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/48cffbd7-f0e4-4308-b8fb-15fed01a4f08-console-serving-cert\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\"" Jul 27 21:08:23.145899 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:08:23.145890 2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48cffbd7-f0e4-4308-b8fb-15fed01a4f08-trusted-ca-bundle\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\"" Jul 27 21:08:23.145899 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:08:23.145899 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6kzt\" (UniqueName: \"kubernetes.io/projected/48cffbd7-f0e4-4308-b8fb-15fed01a4f08-kube-api-access-t6kzt\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\"" Jul 27 21:08:23.145899 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:08:23.145909 2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/48cffbd7-f0e4-4308-b8fb-15fed01a4f08-console-config\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\"" Jul 27 21:08:23.146149 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:08:23.145917 2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/48cffbd7-f0e4-4308-b8fb-15fed01a4f08-service-ca\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\"" Jul 27 21:08:23.146149 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:08:23.145925 2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/48cffbd7-f0e4-4308-b8fb-15fed01a4f08-console-oauth-config\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\"" Jul 27 21:08:23.146149 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:08:23.145935 2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/48cffbd7-f0e4-4308-b8fb-15fed01a4f08-oauth-serving-cert\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\"" Jul 27 21:08:23.656956 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:08:23.656921 2577 generic.go:356] "Generic (PLEG): container finished" podID="48cffbd7-f0e4-4308-b8fb-15fed01a4f08" containerID="7dcabc13eb389647855a63163f7a58955999a55c1680fcb73437b1dbe5f481d8" exitCode=0 Jul 27 21:08:23.657353 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:08:23.656968 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c86bc6544-6qfcp" Jul 27 21:08:23.657353 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:08:23.657011 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c86bc6544-6qfcp" event={"ID":"48cffbd7-f0e4-4308-b8fb-15fed01a4f08","Type":"ContainerDied","Data":"7dcabc13eb389647855a63163f7a58955999a55c1680fcb73437b1dbe5f481d8"} Jul 27 21:08:23.657353 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:08:23.657070 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c86bc6544-6qfcp" event={"ID":"48cffbd7-f0e4-4308-b8fb-15fed01a4f08","Type":"ContainerDied","Data":"6bb3828ee354fb2bd46bdc2acc2b6ccb0ec670dafdb3252a02ad2503b193bea2"} Jul 27 21:08:23.657353 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:08:23.657094 2577 scope.go:117] "RemoveContainer" containerID="7dcabc13eb389647855a63163f7a58955999a55c1680fcb73437b1dbe5f481d8" Jul 27 21:08:23.666182 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:08:23.666165 2577 scope.go:117] "RemoveContainer" containerID="7dcabc13eb389647855a63163f7a58955999a55c1680fcb73437b1dbe5f481d8" Jul 27 21:08:23.666449 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:08:23.666431 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dcabc13eb389647855a63163f7a58955999a55c1680fcb73437b1dbe5f481d8\": container with ID starting with 7dcabc13eb389647855a63163f7a58955999a55c1680fcb73437b1dbe5f481d8 not found: ID does not exist" containerID="7dcabc13eb389647855a63163f7a58955999a55c1680fcb73437b1dbe5f481d8" Jul 27 21:08:23.666484 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:08:23.666459 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dcabc13eb389647855a63163f7a58955999a55c1680fcb73437b1dbe5f481d8"} err="failed to get container status \"7dcabc13eb389647855a63163f7a58955999a55c1680fcb73437b1dbe5f481d8\": rpc error: code = NotFound desc = could not find container \"7dcabc13eb389647855a63163f7a58955999a55c1680fcb73437b1dbe5f481d8\": container with ID starting with 7dcabc13eb389647855a63163f7a58955999a55c1680fcb73437b1dbe5f481d8 not found: ID does not exist" Jul 27 21:08:23.679118 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:08:23.679082 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c86bc6544-6qfcp"] Jul 27 21:08:23.682716 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:08:23.682682 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5c86bc6544-6qfcp"] Jul 27 21:08:24.366038 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:08:24.366001 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48cffbd7-f0e4-4308-b8fb-15fed01a4f08" path="/var/lib/kubelet/pods/48cffbd7-f0e4-4308-b8fb-15fed01a4f08/volumes" Jul 27 21:08:28.225843 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:08:28.225810 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-j5rmn_4812c628-5f41-4e36-aaf6-8d62e4b6a443/console-operator/2.log" Jul 27 21:08:28.230537 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:08:28.230500 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6mhn_1907dc14-1132-4eb9-a5ee-675b43df6e6d/ovn-acl-logging/0.log" Jul 27 21:08:28.234655 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:08:28.234627 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-j5rmn_4812c628-5f41-4e36-aaf6-8d62e4b6a443/console-operator/2.log" Jul 27 21:08:28.237860 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:08:28.237841 2577 kubelet.go:1648] "Image garbage collection succeeded" Jul 27 21:08:28.239403 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:08:28.239384 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6mhn_1907dc14-1132-4eb9-a5ee-675b43df6e6d/ovn-acl-logging/0.log" Jul 27 21:09:37.023957 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:09:37.023849 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mp688"] Jul 27 21:09:37.024509 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:09:37.024217 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48cffbd7-f0e4-4308-b8fb-15fed01a4f08" containerName="console" Jul 27 21:09:37.024509 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:09:37.024232 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="48cffbd7-f0e4-4308-b8fb-15fed01a4f08" containerName="console" Jul 27 21:09:37.024509 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:09:37.024292 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="48cffbd7-f0e4-4308-b8fb-15fed01a4f08" containerName="console" Jul 27 21:09:37.027351 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:09:37.027333 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mp688" Jul 27 21:09:37.030412 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:09:37.030383 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-ntv8j\"" Jul 27 21:09:37.030412 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:09:37.030383 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 27 21:09:37.031392 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:09:37.031369 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 27 21:09:37.042987 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:09:37.042960 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mp688"] Jul 27 21:09:37.147000 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:09:37.146962 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/19bd955c-ae67-446c-b237-0552bd500a22-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mp688\" (UID: \"19bd955c-ae67-446c-b237-0552bd500a22\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mp688" Jul 27 21:09:37.147000 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:09:37.147006 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt9sb\" (UniqueName: \"kubernetes.io/projected/19bd955c-ae67-446c-b237-0552bd500a22-kube-api-access-xt9sb\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mp688\" (UID: \"19bd955c-ae67-446c-b237-0552bd500a22\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mp688" Jul 27 21:09:37.147245 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:09:37.147027 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/19bd955c-ae67-446c-b237-0552bd500a22-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mp688\" (UID: \"19bd955c-ae67-446c-b237-0552bd500a22\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mp688" Jul 27 21:09:37.248070 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:09:37.248031 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/19bd955c-ae67-446c-b237-0552bd500a22-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mp688\" (UID: \"19bd955c-ae67-446c-b237-0552bd500a22\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mp688" Jul 27 21:09:37.248070 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:09:37.248081 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt9sb\" (UniqueName: \"kubernetes.io/projected/19bd955c-ae67-446c-b237-0552bd500a22-kube-api-access-xt9sb\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mp688\" (UID: \"19bd955c-ae67-446c-b237-0552bd500a22\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mp688" Jul 27 21:09:37.248318 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:09:37.248103 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/19bd955c-ae67-446c-b237-0552bd500a22-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mp688\" (UID: \"19bd955c-ae67-446c-b237-0552bd500a22\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mp688" Jul 27 21:09:37.248563 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:09:37.248545 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/19bd955c-ae67-446c-b237-0552bd500a22-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mp688\" (UID: \"19bd955c-ae67-446c-b237-0552bd500a22\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mp688" Jul 27 21:09:37.248624 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:09:37.248561 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/19bd955c-ae67-446c-b237-0552bd500a22-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mp688\" (UID: \"19bd955c-ae67-446c-b237-0552bd500a22\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mp688" Jul 27 21:09:37.256661 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:09:37.256631 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt9sb\" (UniqueName: \"kubernetes.io/projected/19bd955c-ae67-446c-b237-0552bd500a22-kube-api-access-xt9sb\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mp688\" (UID: \"19bd955c-ae67-446c-b237-0552bd500a22\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mp688" Jul 27 21:09:37.336465 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:09:37.336373 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mp688" Jul 27 21:09:37.475548 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:09:37.475510 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mp688"] Jul 27 21:09:37.478675 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:09:37.478643 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19bd955c_ae67_446c_b237_0552bd500a22.slice/crio-2a9272602c812e51c6254198effe0f1873fc07a7c6fad70419b8f7d3281f7a8d WatchSource:0}: Error finding container 2a9272602c812e51c6254198effe0f1873fc07a7c6fad70419b8f7d3281f7a8d: Status 404 returned error can't find the container with id 2a9272602c812e51c6254198effe0f1873fc07a7c6fad70419b8f7d3281f7a8d Jul 27 21:09:37.480500 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:09:37.480484 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 27 21:09:37.940589 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:09:37.940538 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mp688" event={"ID":"19bd955c-ae67-446c-b237-0552bd500a22","Type":"ContainerStarted","Data":"2a9272602c812e51c6254198effe0f1873fc07a7c6fad70419b8f7d3281f7a8d"} Jul 27 21:09:43.974313 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:09:43.974277 2577 generic.go:356] "Generic (PLEG): container finished" podID="19bd955c-ae67-446c-b237-0552bd500a22" containerID="2f1a0bc78c95e5fdf08b997596ddfa8036d986fa2b91b2f2adb28f9d67d1573c" exitCode=0 Jul 27 21:09:43.974734 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:09:43.974354 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mp688" event={"ID":"19bd955c-ae67-446c-b237-0552bd500a22","Type":"ContainerDied","Data":"2f1a0bc78c95e5fdf08b997596ddfa8036d986fa2b91b2f2adb28f9d67d1573c"} Jul 27 21:09:46.988119 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:09:46.988087 2577 generic.go:356] "Generic (PLEG): container finished" podID="19bd955c-ae67-446c-b237-0552bd500a22" containerID="612b54a702ca555235bce6b07b593ea7404d140e1c30134780549e13c862354f" exitCode=0 Jul 27 21:09:46.988512 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:09:46.988177 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mp688" event={"ID":"19bd955c-ae67-446c-b237-0552bd500a22","Type":"ContainerDied","Data":"612b54a702ca555235bce6b07b593ea7404d140e1c30134780549e13c862354f"} Jul 27 21:09:55.026365 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:09:55.026331 2577 generic.go:356] "Generic (PLEG): container finished" podID="19bd955c-ae67-446c-b237-0552bd500a22" containerID="43bddba9b27561eccbbee9d2de5178f7f434d218c9541c1f8fe266c38d7c4e02" exitCode=0 Jul 27 21:09:55.026758 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:09:55.026416 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mp688" event={"ID":"19bd955c-ae67-446c-b237-0552bd500a22","Type":"ContainerDied","Data":"43bddba9b27561eccbbee9d2de5178f7f434d218c9541c1f8fe266c38d7c4e02"} Jul 27 21:09:56.155218 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:09:56.155189 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mp688" Jul 27 21:09:56.207364 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:09:56.207329 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/19bd955c-ae67-446c-b237-0552bd500a22-util\") pod \"19bd955c-ae67-446c-b237-0552bd500a22\" (UID: \"19bd955c-ae67-446c-b237-0552bd500a22\") " Jul 27 21:09:56.207508 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:09:56.207389 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xt9sb\" (UniqueName: \"kubernetes.io/projected/19bd955c-ae67-446c-b237-0552bd500a22-kube-api-access-xt9sb\") pod \"19bd955c-ae67-446c-b237-0552bd500a22\" (UID: \"19bd955c-ae67-446c-b237-0552bd500a22\") " Jul 27 21:09:56.207604 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:09:56.207515 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/19bd955c-ae67-446c-b237-0552bd500a22-bundle\") pod \"19bd955c-ae67-446c-b237-0552bd500a22\" (UID: \"19bd955c-ae67-446c-b237-0552bd500a22\") " Jul 27 21:09:56.208147 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:09:56.208117 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19bd955c-ae67-446c-b237-0552bd500a22-bundle" (OuterVolumeSpecName: "bundle") pod "19bd955c-ae67-446c-b237-0552bd500a22" (UID: "19bd955c-ae67-446c-b237-0552bd500a22"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 21:09:56.209540 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:09:56.209513 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19bd955c-ae67-446c-b237-0552bd500a22-kube-api-access-xt9sb" (OuterVolumeSpecName: "kube-api-access-xt9sb") pod "19bd955c-ae67-446c-b237-0552bd500a22" (UID: "19bd955c-ae67-446c-b237-0552bd500a22"). InnerVolumeSpecName "kube-api-access-xt9sb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 21:09:56.211632 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:09:56.211600 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19bd955c-ae67-446c-b237-0552bd500a22-util" (OuterVolumeSpecName: "util") pod "19bd955c-ae67-446c-b237-0552bd500a22" (UID: "19bd955c-ae67-446c-b237-0552bd500a22"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 21:09:56.308266 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:09:56.308165 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xt9sb\" (UniqueName: \"kubernetes.io/projected/19bd955c-ae67-446c-b237-0552bd500a22-kube-api-access-xt9sb\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\"" Jul 27 21:09:56.308266 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:09:56.308214 2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/19bd955c-ae67-446c-b237-0552bd500a22-bundle\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\"" Jul 27 21:09:56.308266 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:09:56.308225 2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/19bd955c-ae67-446c-b237-0552bd500a22-util\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\"" Jul 27 21:09:57.035634 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:09:57.035605 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mp688" Jul 27 21:09:57.035802 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:09:57.035608 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5mp688" event={"ID":"19bd955c-ae67-446c-b237-0552bd500a22","Type":"ContainerDied","Data":"2a9272602c812e51c6254198effe0f1873fc07a7c6fad70419b8f7d3281f7a8d"} Jul 27 21:09:57.035802 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:09:57.035728 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a9272602c812e51c6254198effe0f1873fc07a7c6fad70419b8f7d3281f7a8d" Jul 27 21:10:00.871922 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:00.871885 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-lmkd8"] Jul 27 21:10:00.872305 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:00.872170 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19bd955c-ae67-446c-b237-0552bd500a22" containerName="util" Jul 27 21:10:00.872305 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:00.872179 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="19bd955c-ae67-446c-b237-0552bd500a22" containerName="util" Jul 27 21:10:00.872305 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:00.872194 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19bd955c-ae67-446c-b237-0552bd500a22" containerName="pull" Jul 27 21:10:00.872305 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:00.872199 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="19bd955c-ae67-446c-b237-0552bd500a22" containerName="pull" Jul 27 21:10:00.872305 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:00.872205 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19bd955c-ae67-446c-b237-0552bd500a22" containerName="extract" Jul 27 21:10:00.872305 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:00.872211 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="19bd955c-ae67-446c-b237-0552bd500a22" containerName="extract" Jul 27 21:10:00.872305 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:00.872258 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="19bd955c-ae67-446c-b237-0552bd500a22" containerName="extract" Jul 27 21:10:00.905096 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:00.905059 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-lmkd8"] Jul 27 21:10:00.905258 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:00.905229 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-lmkd8" Jul 27 21:10:00.908268 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:00.908246 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-szl2w\"" Jul 27 21:10:00.909329 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:00.909309 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\"" Jul 27 21:10:00.909876 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:00.909861 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\"" Jul 27 21:10:00.909972 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:00.909941 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\"" Jul 27 21:10:00.949111 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:00.949069 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/c0c4b8d0-b2d9-42c9-af80-e41d959fbe75-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-lmkd8\" (UID: \"c0c4b8d0-b2d9-42c9-af80-e41d959fbe75\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-lmkd8" Jul 27 21:10:00.949296 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:00.949137 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbz59\" (UniqueName: \"kubernetes.io/projected/c0c4b8d0-b2d9-42c9-af80-e41d959fbe75-kube-api-access-lbz59\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-lmkd8\" (UID: \"c0c4b8d0-b2d9-42c9-af80-e41d959fbe75\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-lmkd8" Jul 27 21:10:01.050309 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:01.050275 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/c0c4b8d0-b2d9-42c9-af80-e41d959fbe75-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-lmkd8\" (UID: \"c0c4b8d0-b2d9-42c9-af80-e41d959fbe75\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-lmkd8" Jul 27 21:10:01.050525 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:01.050321 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbz59\" (UniqueName: \"kubernetes.io/projected/c0c4b8d0-b2d9-42c9-af80-e41d959fbe75-kube-api-access-lbz59\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-lmkd8\" (UID: \"c0c4b8d0-b2d9-42c9-af80-e41d959fbe75\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-lmkd8" Jul 27 21:10:01.052719 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:01.052693 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/c0c4b8d0-b2d9-42c9-af80-e41d959fbe75-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-lmkd8\" (UID: \"c0c4b8d0-b2d9-42c9-af80-e41d959fbe75\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-lmkd8" Jul 27 21:10:01.066469 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:01.066435 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbz59\" (UniqueName: \"kubernetes.io/projected/c0c4b8d0-b2d9-42c9-af80-e41d959fbe75-kube-api-access-lbz59\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-lmkd8\" (UID: \"c0c4b8d0-b2d9-42c9-af80-e41d959fbe75\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-lmkd8" Jul 27 21:10:01.215606 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:01.215494 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-lmkd8" Jul 27 21:10:01.388651 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:01.388622 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-lmkd8"] Jul 27 21:10:01.388909 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:10:01.388883 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0c4b8d0_b2d9_42c9_af80_e41d959fbe75.slice/crio-4a41dc39887cf4459a9396ea47970fe48f441a4fa230a56dd1235dc104c0593f WatchSource:0}: Error finding container 4a41dc39887cf4459a9396ea47970fe48f441a4fa230a56dd1235dc104c0593f: Status 404 returned error can't find the container with id 4a41dc39887cf4459a9396ea47970fe48f441a4fa230a56dd1235dc104c0593f Jul 27 21:10:02.056995 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:02.056960 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-lmkd8" event={"ID":"c0c4b8d0-b2d9-42c9-af80-e41d959fbe75","Type":"ContainerStarted","Data":"4a41dc39887cf4459a9396ea47970fe48f441a4fa230a56dd1235dc104c0593f"} Jul 27 21:10:05.073432 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:05.073391 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-lmkd8" event={"ID":"c0c4b8d0-b2d9-42c9-af80-e41d959fbe75","Type":"ContainerStarted","Data":"37a0afdc5693a4d420c17bbef5823216791ee1e77094ca3cfe889594aeb141a7"} Jul 27 21:10:05.073906 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:05.073532 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-lmkd8" Jul 27 21:10:05.120368 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:05.120315 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-lmkd8" podStartSLOduration=1.611745982 podStartE2EDuration="5.120296821s" totalImagesPullingTime="3.508550839s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:10:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:10:05.115934122 +0000 UTC m=+397.247550345" watchObservedRunningTime="2026-07-27 21:10:05.120296821 +0000 UTC m=+397.251913043" Jul 27 21:10:05.812122 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:05.812091 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-77f648d694-4xxc2"] Jul 27 21:10:05.815305 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:05.815285 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-4xxc2" Jul 27 21:10:05.817903 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:05.817878 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\"" Jul 27 21:10:05.818041 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:05.818021 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\"" Jul 27 21:10:05.818112 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:05.818067 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-9v68g\"" Jul 27 21:10:05.828128 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:05.828092 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-4xxc2"] Jul 27 21:10:05.891719 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:05.891680 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/62841372-9a5a-4ca0-a2df-ef5252683d77-cabundle0\") pod \"keda-operator-77f648d694-4xxc2\" (UID: \"62841372-9a5a-4ca0-a2df-ef5252683d77\") " pod="openshift-keda/keda-operator-77f648d694-4xxc2" Jul 27 21:10:05.891917 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:05.891738 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhfhc\" (UniqueName: \"kubernetes.io/projected/62841372-9a5a-4ca0-a2df-ef5252683d77-kube-api-access-lhfhc\") pod \"keda-operator-77f648d694-4xxc2\" (UID: \"62841372-9a5a-4ca0-a2df-ef5252683d77\") " pod="openshift-keda/keda-operator-77f648d694-4xxc2" Jul 27 21:10:05.891917 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:05.891841 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/62841372-9a5a-4ca0-a2df-ef5252683d77-certificates\") pod \"keda-operator-77f648d694-4xxc2\" (UID: \"62841372-9a5a-4ca0-a2df-ef5252683d77\") " pod="openshift-keda/keda-operator-77f648d694-4xxc2" Jul 27 21:10:05.993153 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:05.993113 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/62841372-9a5a-4ca0-a2df-ef5252683d77-cabundle0\") pod \"keda-operator-77f648d694-4xxc2\" (UID: \"62841372-9a5a-4ca0-a2df-ef5252683d77\") " pod="openshift-keda/keda-operator-77f648d694-4xxc2" Jul 27 21:10:05.993356 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:05.993182 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhfhc\" (UniqueName: \"kubernetes.io/projected/62841372-9a5a-4ca0-a2df-ef5252683d77-kube-api-access-lhfhc\") pod \"keda-operator-77f648d694-4xxc2\" (UID: \"62841372-9a5a-4ca0-a2df-ef5252683d77\") " pod="openshift-keda/keda-operator-77f648d694-4xxc2" Jul 27 21:10:05.993356 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:05.993239 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/62841372-9a5a-4ca0-a2df-ef5252683d77-certificates\") pod \"keda-operator-77f648d694-4xxc2\" (UID: \"62841372-9a5a-4ca0-a2df-ef5252683d77\") " pod="openshift-keda/keda-operator-77f648d694-4xxc2" Jul 27 21:10:05.993477 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:10:05.993388 2577 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found Jul 27 21:10:05.993477 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:10:05.993406 2577 secret.go:281] references non-existent secret key: ca.crt Jul 27 21:10:05.993477 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:10:05.993419 2577 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt Jul 27 21:10:05.993477 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:10:05.993434 2577 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-77f648d694-4xxc2: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt] Jul 27 21:10:05.993650 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:10:05.993500 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62841372-9a5a-4ca0-a2df-ef5252683d77-certificates podName:62841372-9a5a-4ca0-a2df-ef5252683d77 nodeName:}" failed. No retries permitted until 2026-07-27 21:10:06.493478313 +0000 UTC m=+398.625094520 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/62841372-9a5a-4ca0-a2df-ef5252683d77-certificates") pod "keda-operator-77f648d694-4xxc2" (UID: "62841372-9a5a-4ca0-a2df-ef5252683d77") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt] Jul 27 21:10:05.994378 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:05.994352 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/62841372-9a5a-4ca0-a2df-ef5252683d77-cabundle0\") pod \"keda-operator-77f648d694-4xxc2\" (UID: \"62841372-9a5a-4ca0-a2df-ef5252683d77\") " pod="openshift-keda/keda-operator-77f648d694-4xxc2" Jul 27 21:10:06.003252 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:06.003217 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhfhc\" (UniqueName: \"kubernetes.io/projected/62841372-9a5a-4ca0-a2df-ef5252683d77-kube-api-access-lhfhc\") pod \"keda-operator-77f648d694-4xxc2\" (UID: \"62841372-9a5a-4ca0-a2df-ef5252683d77\") " pod="openshift-keda/keda-operator-77f648d694-4xxc2" Jul 27 21:10:06.030674 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:06.030642 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-sdg9d"] Jul 27 21:10:06.033975 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:06.033954 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-sdg9d" Jul 27 21:10:06.037859 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:06.037827 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\"" Jul 27 21:10:06.046268 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:06.046238 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-sdg9d"] Jul 27 21:10:06.094050 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:06.093957 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/abed45ff-d1c3-4af4-8eff-6412cd9eb983-certificates\") pod \"keda-metrics-apiserver-55557df8c-sdg9d\" (UID: \"abed45ff-d1c3-4af4-8eff-6412cd9eb983\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-sdg9d" Jul 27 21:10:06.094050 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:06.094017 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9mzm\" (UniqueName: \"kubernetes.io/projected/abed45ff-d1c3-4af4-8eff-6412cd9eb983-kube-api-access-q9mzm\") pod \"keda-metrics-apiserver-55557df8c-sdg9d\" (UID: \"abed45ff-d1c3-4af4-8eff-6412cd9eb983\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-sdg9d" Jul 27 21:10:06.094050 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:06.094048 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/abed45ff-d1c3-4af4-8eff-6412cd9eb983-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-sdg9d\" (UID: \"abed45ff-d1c3-4af4-8eff-6412cd9eb983\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-sdg9d" Jul 27 21:10:06.194478 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:06.194434 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/abed45ff-d1c3-4af4-8eff-6412cd9eb983-certificates\") pod \"keda-metrics-apiserver-55557df8c-sdg9d\" (UID: \"abed45ff-d1c3-4af4-8eff-6412cd9eb983\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-sdg9d" Jul 27 21:10:06.194725 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:10:06.194613 2577 secret.go:281] references non-existent secret key: tls.crt Jul 27 21:10:06.194725 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:10:06.194635 2577 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt Jul 27 21:10:06.194725 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:10:06.194656 2577 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-55557df8c-sdg9d: references non-existent secret key: tls.crt Jul 27 21:10:06.194725 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:06.194662 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9mzm\" (UniqueName: \"kubernetes.io/projected/abed45ff-d1c3-4af4-8eff-6412cd9eb983-kube-api-access-q9mzm\") pod \"keda-metrics-apiserver-55557df8c-sdg9d\" (UID: \"abed45ff-d1c3-4af4-8eff-6412cd9eb983\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-sdg9d" Jul 27 21:10:06.194725 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:10:06.194704 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/abed45ff-d1c3-4af4-8eff-6412cd9eb983-certificates podName:abed45ff-d1c3-4af4-8eff-6412cd9eb983 nodeName:}" failed. No retries permitted until 2026-07-27 21:10:06.694690218 +0000 UTC m=+398.826306420 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/abed45ff-d1c3-4af4-8eff-6412cd9eb983-certificates") pod "keda-metrics-apiserver-55557df8c-sdg9d" (UID: "abed45ff-d1c3-4af4-8eff-6412cd9eb983") : references non-existent secret key: tls.crt Jul 27 21:10:06.194981 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:06.194765 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/abed45ff-d1c3-4af4-8eff-6412cd9eb983-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-sdg9d\" (UID: \"abed45ff-d1c3-4af4-8eff-6412cd9eb983\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-sdg9d" Jul 27 21:10:06.195180 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:06.195163 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/abed45ff-d1c3-4af4-8eff-6412cd9eb983-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-sdg9d\" (UID: \"abed45ff-d1c3-4af4-8eff-6412cd9eb983\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-sdg9d" Jul 27 21:10:06.208037 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:06.208011 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9mzm\" (UniqueName: \"kubernetes.io/projected/abed45ff-d1c3-4af4-8eff-6412cd9eb983-kube-api-access-q9mzm\") pod \"keda-metrics-apiserver-55557df8c-sdg9d\" (UID: \"abed45ff-d1c3-4af4-8eff-6412cd9eb983\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-sdg9d" Jul 27 21:10:06.498490 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:06.498458 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/62841372-9a5a-4ca0-a2df-ef5252683d77-certificates\") pod \"keda-operator-77f648d694-4xxc2\" (UID: \"62841372-9a5a-4ca0-a2df-ef5252683d77\") " pod="openshift-keda/keda-operator-77f648d694-4xxc2" Jul 27 21:10:06.498688 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:10:06.498597 2577 secret.go:281] references non-existent secret key: ca.crt Jul 27 21:10:06.498688 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:10:06.498612 2577 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt Jul 27 21:10:06.498688 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:10:06.498620 2577 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-77f648d694-4xxc2: references non-existent secret key: ca.crt Jul 27 21:10:06.498688 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:10:06.498668 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62841372-9a5a-4ca0-a2df-ef5252683d77-certificates podName:62841372-9a5a-4ca0-a2df-ef5252683d77 nodeName:}" failed. No retries permitted until 2026-07-27 21:10:07.498654637 +0000 UTC m=+399.630270837 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/62841372-9a5a-4ca0-a2df-ef5252683d77-certificates") pod "keda-operator-77f648d694-4xxc2" (UID: "62841372-9a5a-4ca0-a2df-ef5252683d77") : references non-existent secret key: ca.crt Jul 27 21:10:06.699940 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:06.699903 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/abed45ff-d1c3-4af4-8eff-6412cd9eb983-certificates\") pod \"keda-metrics-apiserver-55557df8c-sdg9d\" (UID: \"abed45ff-d1c3-4af4-8eff-6412cd9eb983\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-sdg9d" Jul 27 21:10:06.700107 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:10:06.700048 2577 secret.go:281] references non-existent secret key: tls.crt Jul 27 21:10:06.700107 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:10:06.700072 2577 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt Jul 27 21:10:06.700107 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:10:06.700092 2577 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-55557df8c-sdg9d: references non-existent secret key: tls.crt Jul 27 21:10:06.700231 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:10:06.700147 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/abed45ff-d1c3-4af4-8eff-6412cd9eb983-certificates podName:abed45ff-d1c3-4af4-8eff-6412cd9eb983 nodeName:}" failed. No retries permitted until 2026-07-27 21:10:07.700131756 +0000 UTC m=+399.831747957 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/abed45ff-d1c3-4af4-8eff-6412cd9eb983-certificates") pod "keda-metrics-apiserver-55557df8c-sdg9d" (UID: "abed45ff-d1c3-4af4-8eff-6412cd9eb983") : references non-existent secret key: tls.crt Jul 27 21:10:06.835871 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:06.835784 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-phnz2"] Jul 27 21:10:06.839179 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:06.839159 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-phnz2" Jul 27 21:10:06.842391 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:06.842367 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\"" Jul 27 21:10:06.853999 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:06.853964 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-phnz2"] Jul 27 21:10:06.901878 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:06.901840 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds9f8\" (UniqueName: \"kubernetes.io/projected/a05f86f5-1b4d-40e0-a246-57bfbc7f6341-kube-api-access-ds9f8\") pod \"keda-admission-56f9798bb4-phnz2\" (UID: \"a05f86f5-1b4d-40e0-a246-57bfbc7f6341\") " pod="openshift-keda/keda-admission-56f9798bb4-phnz2" Jul 27 21:10:06.901878 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:06.901885 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a05f86f5-1b4d-40e0-a246-57bfbc7f6341-certificates\") pod \"keda-admission-56f9798bb4-phnz2\" (UID: \"a05f86f5-1b4d-40e0-a246-57bfbc7f6341\") " pod="openshift-keda/keda-admission-56f9798bb4-phnz2" Jul 27 21:10:07.002915 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:07.002881 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds9f8\" (UniqueName: \"kubernetes.io/projected/a05f86f5-1b4d-40e0-a246-57bfbc7f6341-kube-api-access-ds9f8\") pod \"keda-admission-56f9798bb4-phnz2\" (UID: \"a05f86f5-1b4d-40e0-a246-57bfbc7f6341\") " pod="openshift-keda/keda-admission-56f9798bb4-phnz2" Jul 27 21:10:07.003097 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:07.002923 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a05f86f5-1b4d-40e0-a246-57bfbc7f6341-certificates\") pod \"keda-admission-56f9798bb4-phnz2\" (UID: \"a05f86f5-1b4d-40e0-a246-57bfbc7f6341\") " pod="openshift-keda/keda-admission-56f9798bb4-phnz2" Jul 27 21:10:07.003097 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:10:07.003046 2577 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found Jul 27 21:10:07.003097 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:10:07.003065 2577 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-56f9798bb4-phnz2: secret "keda-admission-webhooks-certs" not found Jul 27 21:10:07.003228 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:10:07.003112 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a05f86f5-1b4d-40e0-a246-57bfbc7f6341-certificates podName:a05f86f5-1b4d-40e0-a246-57bfbc7f6341 nodeName:}" failed. No retries permitted until 2026-07-27 21:10:07.503098918 +0000 UTC m=+399.634715121 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/a05f86f5-1b4d-40e0-a246-57bfbc7f6341-certificates") pod "keda-admission-56f9798bb4-phnz2" (UID: "a05f86f5-1b4d-40e0-a246-57bfbc7f6341") : secret "keda-admission-webhooks-certs" not found Jul 27 21:10:07.020819 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:07.020781 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds9f8\" (UniqueName: \"kubernetes.io/projected/a05f86f5-1b4d-40e0-a246-57bfbc7f6341-kube-api-access-ds9f8\") pod \"keda-admission-56f9798bb4-phnz2\" (UID: \"a05f86f5-1b4d-40e0-a246-57bfbc7f6341\") " pod="openshift-keda/keda-admission-56f9798bb4-phnz2" Jul 27 21:10:07.508221 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:07.508187 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a05f86f5-1b4d-40e0-a246-57bfbc7f6341-certificates\") pod \"keda-admission-56f9798bb4-phnz2\" (UID: \"a05f86f5-1b4d-40e0-a246-57bfbc7f6341\") " pod="openshift-keda/keda-admission-56f9798bb4-phnz2" Jul 27 21:10:07.508664 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:07.508253 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/62841372-9a5a-4ca0-a2df-ef5252683d77-certificates\") pod \"keda-operator-77f648d694-4xxc2\" (UID: \"62841372-9a5a-4ca0-a2df-ef5252683d77\") " pod="openshift-keda/keda-operator-77f648d694-4xxc2" Jul 27 21:10:07.510761 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:07.510735 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a05f86f5-1b4d-40e0-a246-57bfbc7f6341-certificates\") pod \"keda-admission-56f9798bb4-phnz2\" (UID: \"a05f86f5-1b4d-40e0-a246-57bfbc7f6341\") " pod="openshift-keda/keda-admission-56f9798bb4-phnz2" Jul 27 21:10:07.510895 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:07.510774 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/62841372-9a5a-4ca0-a2df-ef5252683d77-certificates\") pod \"keda-operator-77f648d694-4xxc2\" (UID: \"62841372-9a5a-4ca0-a2df-ef5252683d77\") " pod="openshift-keda/keda-operator-77f648d694-4xxc2" Jul 27 21:10:07.625459 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:07.625415 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-4xxc2" Jul 27 21:10:07.711159 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:07.711030 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/abed45ff-d1c3-4af4-8eff-6412cd9eb983-certificates\") pod \"keda-metrics-apiserver-55557df8c-sdg9d\" (UID: \"abed45ff-d1c3-4af4-8eff-6412cd9eb983\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-sdg9d" Jul 27 21:10:07.715092 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:07.715058 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/abed45ff-d1c3-4af4-8eff-6412cd9eb983-certificates\") pod \"keda-metrics-apiserver-55557df8c-sdg9d\" (UID: \"abed45ff-d1c3-4af4-8eff-6412cd9eb983\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-sdg9d" Jul 27 21:10:07.750121 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:07.750089 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-phnz2" Jul 27 21:10:07.792704 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:07.792680 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-4xxc2"] Jul 27 21:10:07.794503 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:10:07.794475 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62841372_9a5a_4ca0_a2df_ef5252683d77.slice/crio-833f45b6942706ba1660e3a610dae758fc5f3982786a5117bfb66e5fec04d79c WatchSource:0}: Error finding container 833f45b6942706ba1660e3a610dae758fc5f3982786a5117bfb66e5fec04d79c: Status 404 returned error can't find the container with id 833f45b6942706ba1660e3a610dae758fc5f3982786a5117bfb66e5fec04d79c Jul 27 21:10:07.844880 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:07.844824 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-sdg9d" Jul 27 21:10:07.905590 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:07.905531 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-phnz2"] Jul 27 21:10:07.913102 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:10:07.913034 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda05f86f5_1b4d_40e0_a246_57bfbc7f6341.slice/crio-45e63dddc7146fc2b26bce6e53cf8a02d74f21189b57bca9a0ea450c77aed1dd WatchSource:0}: Error finding container 45e63dddc7146fc2b26bce6e53cf8a02d74f21189b57bca9a0ea450c77aed1dd: Status 404 returned error can't find the container with id 45e63dddc7146fc2b26bce6e53cf8a02d74f21189b57bca9a0ea450c77aed1dd Jul 27 21:10:07.997805 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:07.997638 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-sdg9d"] Jul 27 21:10:08.000341 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:10:08.000309 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabed45ff_d1c3_4af4_8eff_6412cd9eb983.slice/crio-556e49d67d5d7cd87158e4593b43689204d8495582cffe86367952294f1baaf0 WatchSource:0}: Error finding container 556e49d67d5d7cd87158e4593b43689204d8495582cffe86367952294f1baaf0: Status 404 returned error can't find the container with id 556e49d67d5d7cd87158e4593b43689204d8495582cffe86367952294f1baaf0 Jul 27 21:10:08.091023 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:08.090929 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-sdg9d" event={"ID":"abed45ff-d1c3-4af4-8eff-6412cd9eb983","Type":"ContainerStarted","Data":"556e49d67d5d7cd87158e4593b43689204d8495582cffe86367952294f1baaf0"} Jul 27 21:10:08.092123 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:08.092094 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-phnz2" event={"ID":"a05f86f5-1b4d-40e0-a246-57bfbc7f6341","Type":"ContainerStarted","Data":"45e63dddc7146fc2b26bce6e53cf8a02d74f21189b57bca9a0ea450c77aed1dd"} Jul 27 21:10:08.093143 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:08.093125 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-4xxc2" event={"ID":"62841372-9a5a-4ca0-a2df-ef5252683d77","Type":"ContainerStarted","Data":"833f45b6942706ba1660e3a610dae758fc5f3982786a5117bfb66e5fec04d79c"} Jul 27 21:10:10.109194 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:10.109097 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-phnz2" event={"ID":"a05f86f5-1b4d-40e0-a246-57bfbc7f6341","Type":"ContainerStarted","Data":"607c6a3fff18d7f0918665241626f858a86e31fb692ce384877c5ff8e72b9e19"} Jul 27 21:10:10.109612 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:10.109270 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-56f9798bb4-phnz2" Jul 27 21:10:10.126913 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:10.126795 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-56f9798bb4-phnz2" podStartSLOduration=2.269712512 podStartE2EDuration="4.126761452s" totalImagesPullingTime="1.85704894s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:10:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:10:10.126232295 +0000 UTC m=+402.257848531" watchObservedRunningTime="2026-07-27 21:10:10.126761452 +0000 UTC m=+402.258377674" Jul 27 21:10:12.119226 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:12.119131 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-4xxc2" event={"ID":"62841372-9a5a-4ca0-a2df-ef5252683d77","Type":"ContainerStarted","Data":"9e73cbe9e1988da5e616148449b67ee5f9ed99b9d43c77dc8e165af501019ff5"} Jul 27 21:10:12.119226 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:12.119222 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-77f648d694-4xxc2" Jul 27 21:10:12.120801 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:12.120771 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-sdg9d" event={"ID":"abed45ff-d1c3-4af4-8eff-6412cd9eb983","Type":"ContainerStarted","Data":"1182ba0747a92452494aa075234120b440e6ed346cebf41fe72dd7f980cb0f08"} Jul 27 21:10:12.120938 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:12.120920 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-sdg9d" Jul 27 21:10:12.137329 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:12.137278 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-77f648d694-4xxc2" podStartSLOduration=3.119160354 podStartE2EDuration="7.137261874s" totalImagesPullingTime="4.01810152s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:10:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:10:12.135708879 +0000 UTC m=+404.267325103" watchObservedRunningTime="2026-07-27 21:10:12.137261874 +0000 UTC m=+404.268878096" Jul 27 21:10:12.153267 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:12.153215 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-55557df8c-sdg9d" podStartSLOduration=2.340887297 podStartE2EDuration="6.1531979s" totalImagesPullingTime="3.812310603s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:10:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:10:12.152602791 +0000 UTC m=+404.284219013" watchObservedRunningTime="2026-07-27 21:10:12.1531979 +0000 UTC m=+404.284814124" Jul 27 21:10:23.129719 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:23.129679 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-sdg9d" Jul 27 21:10:26.080599 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:26.080544 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-lmkd8" Jul 27 21:10:31.116330 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:31.116291 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-56f9798bb4-phnz2" Jul 27 21:10:33.128414 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:10:33.128381 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-77f648d694-4xxc2" Jul 27 21:11:26.444682 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:11:26.444643 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-rc7j8"] Jul 27 21:11:26.448003 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:11:26.447985 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-ff96d5959-rc7j8" Jul 27 21:11:26.452325 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:11:26.452301 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\"" Jul 27 21:11:26.452467 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:11:26.452304 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-c9xfd\"" Jul 27 21:11:26.452799 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:11:26.452782 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" Jul 27 21:11:26.452910 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:11:26.452812 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" Jul 27 21:11:26.461354 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:11:26.461327 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-rc7j8"] Jul 27 21:11:26.473231 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:11:26.473192 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-84fc959947-b2mh8"] Jul 27 21:11:26.477242 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:11:26.477210 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-84fc959947-b2mh8" Jul 27 21:11:26.481456 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:11:26.481432 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\"" Jul 27 21:11:26.481707 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:11:26.481688 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-2xggn\"" Jul 27 21:11:26.484535 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:11:26.484509 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-84fc959947-b2mh8"] Jul 27 21:11:26.575731 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:11:26.575689 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr655\" (UniqueName: \"kubernetes.io/projected/bb33cdd6-cc95-467b-9809-48a1e82797e4-kube-api-access-fr655\") pod \"llmisvc-controller-manager-84fc959947-b2mh8\" (UID: \"bb33cdd6-cc95-467b-9809-48a1e82797e4\") " pod="kserve/llmisvc-controller-manager-84fc959947-b2mh8" Jul 27 21:11:26.575901 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:11:26.575766 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bjnh\" (UniqueName: \"kubernetes.io/projected/18a95c92-c423-4ce0-878d-f4b4a8d0638a-kube-api-access-8bjnh\") pod \"kserve-controller-manager-ff96d5959-rc7j8\" (UID: \"18a95c92-c423-4ce0-878d-f4b4a8d0638a\") " pod="kserve/kserve-controller-manager-ff96d5959-rc7j8" Jul 27 21:11:26.575901 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:11:26.575825 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/18a95c92-c423-4ce0-878d-f4b4a8d0638a-cert\") pod \"kserve-controller-manager-ff96d5959-rc7j8\" (UID: \"18a95c92-c423-4ce0-878d-f4b4a8d0638a\") " pod="kserve/kserve-controller-manager-ff96d5959-rc7j8" Jul 27 21:11:26.575901 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:11:26.575850 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bb33cdd6-cc95-467b-9809-48a1e82797e4-cert\") pod \"llmisvc-controller-manager-84fc959947-b2mh8\" (UID: \"bb33cdd6-cc95-467b-9809-48a1e82797e4\") " pod="kserve/llmisvc-controller-manager-84fc959947-b2mh8" Jul 27 21:11:26.677126 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:11:26.677073 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr655\" (UniqueName: \"kubernetes.io/projected/bb33cdd6-cc95-467b-9809-48a1e82797e4-kube-api-access-fr655\") pod \"llmisvc-controller-manager-84fc959947-b2mh8\" (UID: \"bb33cdd6-cc95-467b-9809-48a1e82797e4\") " pod="kserve/llmisvc-controller-manager-84fc959947-b2mh8" Jul 27 21:11:26.677126 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:11:26.677148 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bjnh\" (UniqueName: \"kubernetes.io/projected/18a95c92-c423-4ce0-878d-f4b4a8d0638a-kube-api-access-8bjnh\") pod \"kserve-controller-manager-ff96d5959-rc7j8\" (UID: \"18a95c92-c423-4ce0-878d-f4b4a8d0638a\") " pod="kserve/kserve-controller-manager-ff96d5959-rc7j8" Jul 27 21:11:26.677405 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:11:26.677195 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/18a95c92-c423-4ce0-878d-f4b4a8d0638a-cert\") pod \"kserve-controller-manager-ff96d5959-rc7j8\" (UID: \"18a95c92-c423-4ce0-878d-f4b4a8d0638a\") " pod="kserve/kserve-controller-manager-ff96d5959-rc7j8" Jul 27 21:11:26.677405 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:11:26.677215 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bb33cdd6-cc95-467b-9809-48a1e82797e4-cert\") pod \"llmisvc-controller-manager-84fc959947-b2mh8\" (UID: \"bb33cdd6-cc95-467b-9809-48a1e82797e4\") " pod="kserve/llmisvc-controller-manager-84fc959947-b2mh8" Jul 27 21:11:26.677405 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:11:26.677317 2577 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 27 21:11:26.677405 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:11:26.677371 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb33cdd6-cc95-467b-9809-48a1e82797e4-cert podName:bb33cdd6-cc95-467b-9809-48a1e82797e4 nodeName:}" failed. No retries permitted until 2026-07-27 21:11:27.177355271 +0000 UTC m=+479.308971472 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/bb33cdd6-cc95-467b-9809-48a1e82797e4-cert") pod "llmisvc-controller-manager-84fc959947-b2mh8" (UID: "bb33cdd6-cc95-467b-9809-48a1e82797e4") : secret "llmisvc-webhook-server-cert" not found Jul 27 21:11:26.677405 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:11:26.677317 2577 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 27 21:11:26.677788 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:11:26.677451 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18a95c92-c423-4ce0-878d-f4b4a8d0638a-cert podName:18a95c92-c423-4ce0-878d-f4b4a8d0638a nodeName:}" failed. No retries permitted until 2026-07-27 21:11:27.177437603 +0000 UTC m=+479.309053824 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/18a95c92-c423-4ce0-878d-f4b4a8d0638a-cert") pod "kserve-controller-manager-ff96d5959-rc7j8" (UID: "18a95c92-c423-4ce0-878d-f4b4a8d0638a") : secret "kserve-webhook-server-cert" not found Jul 27 21:11:26.691308 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:11:26.691275 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bjnh\" (UniqueName: \"kubernetes.io/projected/18a95c92-c423-4ce0-878d-f4b4a8d0638a-kube-api-access-8bjnh\") pod \"kserve-controller-manager-ff96d5959-rc7j8\" (UID: \"18a95c92-c423-4ce0-878d-f4b4a8d0638a\") " pod="kserve/kserve-controller-manager-ff96d5959-rc7j8" Jul 27 21:11:26.696888 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:11:26.696820 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr655\" (UniqueName: \"kubernetes.io/projected/bb33cdd6-cc95-467b-9809-48a1e82797e4-kube-api-access-fr655\") pod \"llmisvc-controller-manager-84fc959947-b2mh8\" (UID: \"bb33cdd6-cc95-467b-9809-48a1e82797e4\") " pod="kserve/llmisvc-controller-manager-84fc959947-b2mh8" Jul 27 21:11:27.182487 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:11:27.182450 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/18a95c92-c423-4ce0-878d-f4b4a8d0638a-cert\") pod \"kserve-controller-manager-ff96d5959-rc7j8\" (UID: \"18a95c92-c423-4ce0-878d-f4b4a8d0638a\") " pod="kserve/kserve-controller-manager-ff96d5959-rc7j8" Jul 27 21:11:27.182487 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:11:27.182489 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bb33cdd6-cc95-467b-9809-48a1e82797e4-cert\") pod \"llmisvc-controller-manager-84fc959947-b2mh8\" (UID: \"bb33cdd6-cc95-467b-9809-48a1e82797e4\") " pod="kserve/llmisvc-controller-manager-84fc959947-b2mh8" Jul 27 21:11:27.185078 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:11:27.185040 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bb33cdd6-cc95-467b-9809-48a1e82797e4-cert\") pod \"llmisvc-controller-manager-84fc959947-b2mh8\" (UID: \"bb33cdd6-cc95-467b-9809-48a1e82797e4\") " pod="kserve/llmisvc-controller-manager-84fc959947-b2mh8" Jul 27 21:11:27.185212 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:11:27.185095 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/18a95c92-c423-4ce0-878d-f4b4a8d0638a-cert\") pod \"kserve-controller-manager-ff96d5959-rc7j8\" (UID: \"18a95c92-c423-4ce0-878d-f4b4a8d0638a\") " pod="kserve/kserve-controller-manager-ff96d5959-rc7j8" Jul 27 21:11:27.358371 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:11:27.358328 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-ff96d5959-rc7j8" Jul 27 21:11:27.389891 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:11:27.389855 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-84fc959947-b2mh8" Jul 27 21:11:27.509114 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:11:27.509078 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-rc7j8"] Jul 27 21:11:27.514849 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:11:27.514803 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18a95c92_c423_4ce0_878d_f4b4a8d0638a.slice/crio-8d5d9a56712b23822fc99ad827ea8237a8e5a47b7dfdecaaa5a6fde4c3dcfbae WatchSource:0}: Error finding container 8d5d9a56712b23822fc99ad827ea8237a8e5a47b7dfdecaaa5a6fde4c3dcfbae: Status 404 returned error can't find the container with id 8d5d9a56712b23822fc99ad827ea8237a8e5a47b7dfdecaaa5a6fde4c3dcfbae Jul 27 21:11:27.536258 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:11:27.536232 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-84fc959947-b2mh8"] Jul 27 21:11:27.538519 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:11:27.538490 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podbb33cdd6_cc95_467b_9809_48a1e82797e4.slice/crio-997889cd1c11953e535cebeece5a0702b7e78224b56bfd58e1350b21d9163aa8 WatchSource:0}: Error finding container 997889cd1c11953e535cebeece5a0702b7e78224b56bfd58e1350b21d9163aa8: Status 404 returned error can't find the container with id 997889cd1c11953e535cebeece5a0702b7e78224b56bfd58e1350b21d9163aa8 Jul 27 21:11:28.469996 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:11:28.469948 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-ff96d5959-rc7j8" event={"ID":"18a95c92-c423-4ce0-878d-f4b4a8d0638a","Type":"ContainerStarted","Data":"8d5d9a56712b23822fc99ad827ea8237a8e5a47b7dfdecaaa5a6fde4c3dcfbae"} Jul 27 21:11:28.473151 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:11:28.473054 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-84fc959947-b2mh8" event={"ID":"bb33cdd6-cc95-467b-9809-48a1e82797e4","Type":"ContainerStarted","Data":"997889cd1c11953e535cebeece5a0702b7e78224b56bfd58e1350b21d9163aa8"} Jul 27 21:11:31.499622 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:11:31.499552 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-ff96d5959-rc7j8" event={"ID":"18a95c92-c423-4ce0-878d-f4b4a8d0638a","Type":"ContainerStarted","Data":"e94334d0dcfeff922b78182566203658fbce57dc532240bc0ffc4550442e3b0a"} Jul 27 21:11:31.500081 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:11:31.499658 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-ff96d5959-rc7j8" Jul 27 21:11:31.501021 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:11:31.500997 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-84fc959947-b2mh8" event={"ID":"bb33cdd6-cc95-467b-9809-48a1e82797e4","Type":"ContainerStarted","Data":"741d2fff23b4898afdeaf9fabc5543f916d145ae67533be3f9788c6c6ef590cd"} Jul 27 21:11:31.501146 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:11:31.501133 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-84fc959947-b2mh8" Jul 27 21:11:31.518805 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:11:31.518750 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-ff96d5959-rc7j8" podStartSLOduration=2.063781442 podStartE2EDuration="5.518733437s" totalImagesPullingTime="3.454951995s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:11:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:11:31.516814002 +0000 UTC m=+483.648430225" watchObservedRunningTime="2026-07-27 21:11:31.518733437 +0000 UTC m=+483.650349659" Jul 27 21:11:31.534457 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:11:31.534404 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-84fc959947-b2mh8" podStartSLOduration=2.101301267 podStartE2EDuration="5.53438564s" totalImagesPullingTime="3.433084373s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:11:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:11:31.533364276 +0000 UTC m=+483.664980500" watchObservedRunningTime="2026-07-27 21:11:31.53438564 +0000 UTC m=+483.666001864" Jul 27 21:12:02.508703 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:02.508671 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-84fc959947-b2mh8" Jul 27 21:12:02.511694 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:02.511669 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-ff96d5959-rc7j8" Jul 27 21:12:06.943039 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:06.943000 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5ffbdbcc9c-sjlcr"] Jul 27 21:12:06.950157 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:06.950129 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5ffbdbcc9c-sjlcr" Jul 27 21:12:06.970664 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:06.970638 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5ffbdbcc9c-sjlcr"] Jul 27 21:12:07.024896 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:07.024856 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f4a44d7-6d72-481e-86fe-8040a3eb0379-trusted-ca-bundle\") pod \"console-5ffbdbcc9c-sjlcr\" (UID: \"8f4a44d7-6d72-481e-86fe-8040a3eb0379\") " pod="openshift-console/console-5ffbdbcc9c-sjlcr" Jul 27 21:12:07.025078 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:07.024956 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f4a44d7-6d72-481e-86fe-8040a3eb0379-service-ca\") pod \"console-5ffbdbcc9c-sjlcr\" (UID: \"8f4a44d7-6d72-481e-86fe-8040a3eb0379\") " pod="openshift-console/console-5ffbdbcc9c-sjlcr" Jul 27 21:12:07.025078 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:07.024983 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f4a44d7-6d72-481e-86fe-8040a3eb0379-console-oauth-config\") pod \"console-5ffbdbcc9c-sjlcr\" (UID: \"8f4a44d7-6d72-481e-86fe-8040a3eb0379\") " pod="openshift-console/console-5ffbdbcc9c-sjlcr" Jul 27 21:12:07.025078 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:07.025023 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f4a44d7-6d72-481e-86fe-8040a3eb0379-oauth-serving-cert\") pod \"console-5ffbdbcc9c-sjlcr\" (UID: \"8f4a44d7-6d72-481e-86fe-8040a3eb0379\") " pod="openshift-console/console-5ffbdbcc9c-sjlcr" Jul 27 21:12:07.025078 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:07.025068 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f4a44d7-6d72-481e-86fe-8040a3eb0379-console-serving-cert\") pod \"console-5ffbdbcc9c-sjlcr\" (UID: \"8f4a44d7-6d72-481e-86fe-8040a3eb0379\") " pod="openshift-console/console-5ffbdbcc9c-sjlcr" Jul 27 21:12:07.025248 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:07.025104 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f4a44d7-6d72-481e-86fe-8040a3eb0379-console-config\") pod \"console-5ffbdbcc9c-sjlcr\" (UID: \"8f4a44d7-6d72-481e-86fe-8040a3eb0379\") " pod="openshift-console/console-5ffbdbcc9c-sjlcr" Jul 27 21:12:07.025248 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:07.025228 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rbxn\" (UniqueName: \"kubernetes.io/projected/8f4a44d7-6d72-481e-86fe-8040a3eb0379-kube-api-access-8rbxn\") pod \"console-5ffbdbcc9c-sjlcr\" (UID: \"8f4a44d7-6d72-481e-86fe-8040a3eb0379\") " pod="openshift-console/console-5ffbdbcc9c-sjlcr" Jul 27 21:12:07.125904 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:07.125866 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f4a44d7-6d72-481e-86fe-8040a3eb0379-console-config\") pod \"console-5ffbdbcc9c-sjlcr\" (UID: \"8f4a44d7-6d72-481e-86fe-8040a3eb0379\") " pod="openshift-console/console-5ffbdbcc9c-sjlcr" Jul 27 21:12:07.125904 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:07.125912 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rbxn\" (UniqueName: \"kubernetes.io/projected/8f4a44d7-6d72-481e-86fe-8040a3eb0379-kube-api-access-8rbxn\") pod \"console-5ffbdbcc9c-sjlcr\" (UID: \"8f4a44d7-6d72-481e-86fe-8040a3eb0379\") " pod="openshift-console/console-5ffbdbcc9c-sjlcr" Jul 27 21:12:07.126133 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:07.125932 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f4a44d7-6d72-481e-86fe-8040a3eb0379-trusted-ca-bundle\") pod \"console-5ffbdbcc9c-sjlcr\" (UID: \"8f4a44d7-6d72-481e-86fe-8040a3eb0379\") " pod="openshift-console/console-5ffbdbcc9c-sjlcr" Jul 27 21:12:07.126133 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:07.125973 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f4a44d7-6d72-481e-86fe-8040a3eb0379-service-ca\") pod \"console-5ffbdbcc9c-sjlcr\" (UID: \"8f4a44d7-6d72-481e-86fe-8040a3eb0379\") " pod="openshift-console/console-5ffbdbcc9c-sjlcr" Jul 27 21:12:07.126133 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:07.125991 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f4a44d7-6d72-481e-86fe-8040a3eb0379-console-oauth-config\") pod \"console-5ffbdbcc9c-sjlcr\" (UID: \"8f4a44d7-6d72-481e-86fe-8040a3eb0379\") " pod="openshift-console/console-5ffbdbcc9c-sjlcr" Jul 27 21:12:07.126133 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:07.126023 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f4a44d7-6d72-481e-86fe-8040a3eb0379-oauth-serving-cert\") pod \"console-5ffbdbcc9c-sjlcr\" (UID: \"8f4a44d7-6d72-481e-86fe-8040a3eb0379\") " pod="openshift-console/console-5ffbdbcc9c-sjlcr" Jul 27 21:12:07.126133 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:07.126053 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f4a44d7-6d72-481e-86fe-8040a3eb0379-console-serving-cert\") pod \"console-5ffbdbcc9c-sjlcr\" (UID: \"8f4a44d7-6d72-481e-86fe-8040a3eb0379\") " pod="openshift-console/console-5ffbdbcc9c-sjlcr" Jul 27 21:12:07.126852 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:07.126823 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f4a44d7-6d72-481e-86fe-8040a3eb0379-service-ca\") pod \"console-5ffbdbcc9c-sjlcr\" (UID: \"8f4a44d7-6d72-481e-86fe-8040a3eb0379\") " pod="openshift-console/console-5ffbdbcc9c-sjlcr" Jul 27 21:12:07.126852 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:07.126845 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f4a44d7-6d72-481e-86fe-8040a3eb0379-console-config\") pod \"console-5ffbdbcc9c-sjlcr\" (UID: \"8f4a44d7-6d72-481e-86fe-8040a3eb0379\") " pod="openshift-console/console-5ffbdbcc9c-sjlcr" Jul 27 21:12:07.127044 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:07.126850 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f4a44d7-6d72-481e-86fe-8040a3eb0379-oauth-serving-cert\") pod \"console-5ffbdbcc9c-sjlcr\" (UID: \"8f4a44d7-6d72-481e-86fe-8040a3eb0379\") " pod="openshift-console/console-5ffbdbcc9c-sjlcr" Jul 27 21:12:07.127044 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:07.126900 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f4a44d7-6d72-481e-86fe-8040a3eb0379-trusted-ca-bundle\") pod \"console-5ffbdbcc9c-sjlcr\" (UID: \"8f4a44d7-6d72-481e-86fe-8040a3eb0379\") " pod="openshift-console/console-5ffbdbcc9c-sjlcr" Jul 27 21:12:07.128738 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:07.128710 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f4a44d7-6d72-481e-86fe-8040a3eb0379-console-serving-cert\") pod \"console-5ffbdbcc9c-sjlcr\" (UID: \"8f4a44d7-6d72-481e-86fe-8040a3eb0379\") " pod="openshift-console/console-5ffbdbcc9c-sjlcr" Jul 27 21:12:07.128738 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:07.128722 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f4a44d7-6d72-481e-86fe-8040a3eb0379-console-oauth-config\") pod \"console-5ffbdbcc9c-sjlcr\" (UID: \"8f4a44d7-6d72-481e-86fe-8040a3eb0379\") " pod="openshift-console/console-5ffbdbcc9c-sjlcr" Jul 27 21:12:07.135713 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:07.135693 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rbxn\" (UniqueName: \"kubernetes.io/projected/8f4a44d7-6d72-481e-86fe-8040a3eb0379-kube-api-access-8rbxn\") pod \"console-5ffbdbcc9c-sjlcr\" (UID: \"8f4a44d7-6d72-481e-86fe-8040a3eb0379\") " pod="openshift-console/console-5ffbdbcc9c-sjlcr" Jul 27 21:12:07.259180 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:07.259137 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5ffbdbcc9c-sjlcr" Jul 27 21:12:07.404785 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:07.404757 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5ffbdbcc9c-sjlcr"] Jul 27 21:12:07.407706 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:12:07.407674 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f4a44d7_6d72_481e_86fe_8040a3eb0379.slice/crio-abb0c5979bc1c319dd44344b32d7e977068882228d1cc2134710780ecf16ca07 WatchSource:0}: Error finding container abb0c5979bc1c319dd44344b32d7e977068882228d1cc2134710780ecf16ca07: Status 404 returned error can't find the container with id abb0c5979bc1c319dd44344b32d7e977068882228d1cc2134710780ecf16ca07 Jul 27 21:12:07.663109 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:07.663017 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5ffbdbcc9c-sjlcr" event={"ID":"8f4a44d7-6d72-481e-86fe-8040a3eb0379","Type":"ContainerStarted","Data":"cec9fecfb2326b8d6d1016cfbc2431c320e99250d7f40e9ccfdcc9a6a9997336"} Jul 27 21:12:07.663109 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:07.663070 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5ffbdbcc9c-sjlcr" event={"ID":"8f4a44d7-6d72-481e-86fe-8040a3eb0379","Type":"ContainerStarted","Data":"abb0c5979bc1c319dd44344b32d7e977068882228d1cc2134710780ecf16ca07"} Jul 27 21:12:07.690526 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:07.690459 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5ffbdbcc9c-sjlcr" podStartSLOduration=1.690437074 podStartE2EDuration="1.690437074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:12:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:12:07.68274442 +0000 UTC m=+519.814360643" watchObservedRunningTime="2026-07-27 21:12:07.690437074 +0000 UTC m=+519.822053298" Jul 27 21:12:11.200358 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:11.200323 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-rc7j8"] Jul 27 21:12:11.200819 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:11.200518 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-ff96d5959-rc7j8" podUID="18a95c92-c423-4ce0-878d-f4b4a8d0638a" containerName="manager" containerID="cri-o://e94334d0dcfeff922b78182566203658fbce57dc532240bc0ffc4550442e3b0a" gracePeriod=10 Jul 27 21:12:11.224467 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:11.224439 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-8l6t7"] Jul 27 21:12:11.227774 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:11.227756 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-ff96d5959-8l6t7" Jul 27 21:12:11.237314 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:11.237281 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-8l6t7"] Jul 27 21:12:11.365711 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:11.365674 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvk9m\" (UniqueName: \"kubernetes.io/projected/2a3c43a5-306b-4c3d-a6de-c48ba29acead-kube-api-access-cvk9m\") pod \"kserve-controller-manager-ff96d5959-8l6t7\" (UID: \"2a3c43a5-306b-4c3d-a6de-c48ba29acead\") " pod="kserve/kserve-controller-manager-ff96d5959-8l6t7" Jul 27 21:12:11.365893 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:11.365769 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2a3c43a5-306b-4c3d-a6de-c48ba29acead-cert\") pod \"kserve-controller-manager-ff96d5959-8l6t7\" (UID: \"2a3c43a5-306b-4c3d-a6de-c48ba29acead\") " pod="kserve/kserve-controller-manager-ff96d5959-8l6t7" Jul 27 21:12:11.454053 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:11.453985 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-ff96d5959-rc7j8" Jul 27 21:12:11.466644 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:11.466607 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvk9m\" (UniqueName: \"kubernetes.io/projected/2a3c43a5-306b-4c3d-a6de-c48ba29acead-kube-api-access-cvk9m\") pod \"kserve-controller-manager-ff96d5959-8l6t7\" (UID: \"2a3c43a5-306b-4c3d-a6de-c48ba29acead\") " pod="kserve/kserve-controller-manager-ff96d5959-8l6t7" Jul 27 21:12:11.466835 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:11.466670 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2a3c43a5-306b-4c3d-a6de-c48ba29acead-cert\") pod \"kserve-controller-manager-ff96d5959-8l6t7\" (UID: \"2a3c43a5-306b-4c3d-a6de-c48ba29acead\") " pod="kserve/kserve-controller-manager-ff96d5959-8l6t7" Jul 27 21:12:11.469031 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:11.468997 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2a3c43a5-306b-4c3d-a6de-c48ba29acead-cert\") pod \"kserve-controller-manager-ff96d5959-8l6t7\" (UID: \"2a3c43a5-306b-4c3d-a6de-c48ba29acead\") " pod="kserve/kserve-controller-manager-ff96d5959-8l6t7" Jul 27 21:12:11.475557 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:11.475529 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvk9m\" (UniqueName: \"kubernetes.io/projected/2a3c43a5-306b-4c3d-a6de-c48ba29acead-kube-api-access-cvk9m\") pod \"kserve-controller-manager-ff96d5959-8l6t7\" (UID: \"2a3c43a5-306b-4c3d-a6de-c48ba29acead\") " pod="kserve/kserve-controller-manager-ff96d5959-8l6t7" Jul 27 21:12:11.567080 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:11.567047 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8bjnh\" (UniqueName: \"kubernetes.io/projected/18a95c92-c423-4ce0-878d-f4b4a8d0638a-kube-api-access-8bjnh\") pod \"18a95c92-c423-4ce0-878d-f4b4a8d0638a\" (UID: \"18a95c92-c423-4ce0-878d-f4b4a8d0638a\") " Jul 27 21:12:11.567238 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:11.567103 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/18a95c92-c423-4ce0-878d-f4b4a8d0638a-cert\") pod \"18a95c92-c423-4ce0-878d-f4b4a8d0638a\" (UID: \"18a95c92-c423-4ce0-878d-f4b4a8d0638a\") " Jul 27 21:12:11.569394 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:11.569361 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18a95c92-c423-4ce0-878d-f4b4a8d0638a-cert" (OuterVolumeSpecName: "cert") pod "18a95c92-c423-4ce0-878d-f4b4a8d0638a" (UID: "18a95c92-c423-4ce0-878d-f4b4a8d0638a"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 21:12:11.569584 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:11.569374 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18a95c92-c423-4ce0-878d-f4b4a8d0638a-kube-api-access-8bjnh" (OuterVolumeSpecName: "kube-api-access-8bjnh") pod "18a95c92-c423-4ce0-878d-f4b4a8d0638a" (UID: "18a95c92-c423-4ce0-878d-f4b4a8d0638a"). InnerVolumeSpecName "kube-api-access-8bjnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 21:12:11.585717 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:11.585686 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-ff96d5959-8l6t7" Jul 27 21:12:11.667793 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:11.667755 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bjnh\" (UniqueName: \"kubernetes.io/projected/18a95c92-c423-4ce0-878d-f4b4a8d0638a-kube-api-access-8bjnh\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\"" Jul 27 21:12:11.667793 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:11.667792 2577 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/18a95c92-c423-4ce0-878d-f4b4a8d0638a-cert\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\"" Jul 27 21:12:11.694346 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:11.694307 2577 generic.go:356] "Generic (PLEG): container finished" podID="18a95c92-c423-4ce0-878d-f4b4a8d0638a" containerID="e94334d0dcfeff922b78182566203658fbce57dc532240bc0ffc4550442e3b0a" exitCode=0 Jul 27 21:12:11.694503 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:11.694377 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-ff96d5959-rc7j8" Jul 27 21:12:11.694503 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:11.694383 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-ff96d5959-rc7j8" event={"ID":"18a95c92-c423-4ce0-878d-f4b4a8d0638a","Type":"ContainerDied","Data":"e94334d0dcfeff922b78182566203658fbce57dc532240bc0ffc4550442e3b0a"} Jul 27 21:12:11.694635 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:11.694583 2577 scope.go:117] "RemoveContainer" containerID="e94334d0dcfeff922b78182566203658fbce57dc532240bc0ffc4550442e3b0a" Jul 27 21:12:11.694683 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:11.694491 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-ff96d5959-rc7j8" event={"ID":"18a95c92-c423-4ce0-878d-f4b4a8d0638a","Type":"ContainerDied","Data":"8d5d9a56712b23822fc99ad827ea8237a8e5a47b7dfdecaaa5a6fde4c3dcfbae"} Jul 27 21:12:11.703096 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:11.703078 2577 scope.go:117] "RemoveContainer" containerID="e94334d0dcfeff922b78182566203658fbce57dc532240bc0ffc4550442e3b0a" Jul 27 21:12:11.703414 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:12:11.703387 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e94334d0dcfeff922b78182566203658fbce57dc532240bc0ffc4550442e3b0a\": container with ID starting with e94334d0dcfeff922b78182566203658fbce57dc532240bc0ffc4550442e3b0a not found: ID does not exist" containerID="e94334d0dcfeff922b78182566203658fbce57dc532240bc0ffc4550442e3b0a" Jul 27 21:12:11.703517 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:11.703426 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e94334d0dcfeff922b78182566203658fbce57dc532240bc0ffc4550442e3b0a"} err="failed to get container status \"e94334d0dcfeff922b78182566203658fbce57dc532240bc0ffc4550442e3b0a\": rpc error: code = NotFound desc = could not find container \"e94334d0dcfeff922b78182566203658fbce57dc532240bc0ffc4550442e3b0a\": container with ID starting with e94334d0dcfeff922b78182566203658fbce57dc532240bc0ffc4550442e3b0a not found: ID does not exist" Jul 27 21:12:11.719201 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:11.719168 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-rc7j8"] Jul 27 21:12:11.721027 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:11.721004 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-rc7j8"] Jul 27 21:12:11.725363 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:11.725342 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-8l6t7"] Jul 27 21:12:11.727998 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:12:11.727974 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a3c43a5_306b_4c3d_a6de_c48ba29acead.slice/crio-fe2be403adabe866d9aa04ce5b5818a220f2749c707297ae9352a1ffc5eac674 WatchSource:0}: Error finding container fe2be403adabe866d9aa04ce5b5818a220f2749c707297ae9352a1ffc5eac674: Status 404 returned error can't find the container with id fe2be403adabe866d9aa04ce5b5818a220f2749c707297ae9352a1ffc5eac674 Jul 27 21:12:12.368001 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:12.367924 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18a95c92-c423-4ce0-878d-f4b4a8d0638a" path="/var/lib/kubelet/pods/18a95c92-c423-4ce0-878d-f4b4a8d0638a/volumes" Jul 27 21:12:12.702116 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:12.702026 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-ff96d5959-8l6t7" event={"ID":"2a3c43a5-306b-4c3d-a6de-c48ba29acead","Type":"ContainerStarted","Data":"be8377c663e00b6db95f785ae54a01798bfa0c37d80fc033cef550004c85956e"} Jul 27 21:12:12.702116 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:12.702076 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-ff96d5959-8l6t7" event={"ID":"2a3c43a5-306b-4c3d-a6de-c48ba29acead","Type":"ContainerStarted","Data":"fe2be403adabe866d9aa04ce5b5818a220f2749c707297ae9352a1ffc5eac674"} Jul 27 21:12:12.702333 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:12.702123 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-ff96d5959-8l6t7" Jul 27 21:12:12.718390 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:12.718334 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-ff96d5959-8l6t7" podStartSLOduration=1.392457385 podStartE2EDuration="1.718316775s" totalImagesPullingTime="325.85939ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:12:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:12:12.717516504 +0000 UTC m=+524.849132729" watchObservedRunningTime="2026-07-27 21:12:12.718316775 +0000 UTC m=+524.849932997" Jul 27 21:12:17.259909 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:17.259863 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5ffbdbcc9c-sjlcr" Jul 27 21:12:17.260388 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:17.259957 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5ffbdbcc9c-sjlcr" Jul 27 21:12:17.264869 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:17.264842 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5ffbdbcc9c-sjlcr" Jul 27 21:12:17.727514 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:17.727483 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5ffbdbcc9c-sjlcr" Jul 27 21:12:17.780407 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:17.780365 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-dcbd5694-ntknm"] Jul 27 21:12:42.806681 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:42.806616 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-dcbd5694-ntknm" podUID="ef24c526-4136-46ba-973e-b570b043aff6" containerName="console" containerID="cri-o://1c0cac8c8e93c336b9ce4b9f35ba0531364a4a4da6679b29e397f486330cf060" gracePeriod=15 Jul 27 21:12:43.054916 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:43.054889 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dcbd5694-ntknm" Jul 27 21:12:43.132202 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:43.132117 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ef24c526-4136-46ba-973e-b570b043aff6-console-oauth-config\") pod \"ef24c526-4136-46ba-973e-b570b043aff6\" (UID: \"ef24c526-4136-46ba-973e-b570b043aff6\") " Jul 27 21:12:43.132202 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:43.132153 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef24c526-4136-46ba-973e-b570b043aff6-trusted-ca-bundle\") pod \"ef24c526-4136-46ba-973e-b570b043aff6\" (UID: \"ef24c526-4136-46ba-973e-b570b043aff6\") " Jul 27 21:12:43.132202 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:43.132193 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ef24c526-4136-46ba-973e-b570b043aff6-oauth-serving-cert\") pod \"ef24c526-4136-46ba-973e-b570b043aff6\" (UID: \"ef24c526-4136-46ba-973e-b570b043aff6\") " Jul 27 21:12:43.132479 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:43.132211 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ef24c526-4136-46ba-973e-b570b043aff6-console-serving-cert\") pod \"ef24c526-4136-46ba-973e-b570b043aff6\" (UID: \"ef24c526-4136-46ba-973e-b570b043aff6\") " Jul 27 21:12:43.132479 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:43.132250 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ef24c526-4136-46ba-973e-b570b043aff6-console-config\") pod \"ef24c526-4136-46ba-973e-b570b043aff6\" (UID: \"ef24c526-4136-46ba-973e-b570b043aff6\") " Jul 27 21:12:43.132479 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:43.132298 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbmb6\" (UniqueName: \"kubernetes.io/projected/ef24c526-4136-46ba-973e-b570b043aff6-kube-api-access-dbmb6\") pod \"ef24c526-4136-46ba-973e-b570b043aff6\" (UID: \"ef24c526-4136-46ba-973e-b570b043aff6\") " Jul 27 21:12:43.132479 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:43.132369 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ef24c526-4136-46ba-973e-b570b043aff6-service-ca\") pod \"ef24c526-4136-46ba-973e-b570b043aff6\" (UID: \"ef24c526-4136-46ba-973e-b570b043aff6\") " Jul 27 21:12:43.132725 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:43.132701 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef24c526-4136-46ba-973e-b570b043aff6-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ef24c526-4136-46ba-973e-b570b043aff6" (UID: "ef24c526-4136-46ba-973e-b570b043aff6"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 21:12:43.132782 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:43.132716 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef24c526-4136-46ba-973e-b570b043aff6-console-config" (OuterVolumeSpecName: "console-config") pod "ef24c526-4136-46ba-973e-b570b043aff6" (UID: "ef24c526-4136-46ba-973e-b570b043aff6"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 21:12:43.132850 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:43.132726 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef24c526-4136-46ba-973e-b570b043aff6-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ef24c526-4136-46ba-973e-b570b043aff6" (UID: "ef24c526-4136-46ba-973e-b570b043aff6"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 21:12:43.132900 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:43.132887 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef24c526-4136-46ba-973e-b570b043aff6-service-ca" (OuterVolumeSpecName: "service-ca") pod "ef24c526-4136-46ba-973e-b570b043aff6" (UID: "ef24c526-4136-46ba-973e-b570b043aff6"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 21:12:43.134340 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:43.134313 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef24c526-4136-46ba-973e-b570b043aff6-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ef24c526-4136-46ba-973e-b570b043aff6" (UID: "ef24c526-4136-46ba-973e-b570b043aff6"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 21:12:43.134720 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:43.134694 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef24c526-4136-46ba-973e-b570b043aff6-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ef24c526-4136-46ba-973e-b570b043aff6" (UID: "ef24c526-4136-46ba-973e-b570b043aff6"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 21:12:43.134805 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:43.134710 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef24c526-4136-46ba-973e-b570b043aff6-kube-api-access-dbmb6" (OuterVolumeSpecName: "kube-api-access-dbmb6") pod "ef24c526-4136-46ba-973e-b570b043aff6" (UID: "ef24c526-4136-46ba-973e-b570b043aff6"). InnerVolumeSpecName "kube-api-access-dbmb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 21:12:43.233797 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:43.233744 2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ef24c526-4136-46ba-973e-b570b043aff6-service-ca\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\"" Jul 27 21:12:43.233797 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:43.233789 2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ef24c526-4136-46ba-973e-b570b043aff6-console-oauth-config\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\"" Jul 27 21:12:43.233797 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:43.233801 2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef24c526-4136-46ba-973e-b570b043aff6-trusted-ca-bundle\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\"" Jul 27 21:12:43.233797 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:43.233811 2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ef24c526-4136-46ba-973e-b570b043aff6-oauth-serving-cert\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\"" Jul 27 21:12:43.233797 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:43.233820 2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ef24c526-4136-46ba-973e-b570b043aff6-console-serving-cert\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\"" Jul 27 21:12:43.234107 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:43.233828 2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ef24c526-4136-46ba-973e-b570b043aff6-console-config\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\"" Jul 27 21:12:43.234107 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:43.233837 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbmb6\" (UniqueName: \"kubernetes.io/projected/ef24c526-4136-46ba-973e-b570b043aff6-kube-api-access-dbmb6\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\"" Jul 27 21:12:43.712440 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:43.712401 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-ff96d5959-8l6t7" Jul 27 21:12:43.840331 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:43.840294 2577 generic.go:356] "Generic (PLEG): container finished" podID="ef24c526-4136-46ba-973e-b570b043aff6" containerID="1c0cac8c8e93c336b9ce4b9f35ba0531364a4a4da6679b29e397f486330cf060" exitCode=0 Jul 27 21:12:43.840331 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:43.840331 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dcbd5694-ntknm" Jul 27 21:12:43.840811 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:43.840361 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dcbd5694-ntknm" event={"ID":"ef24c526-4136-46ba-973e-b570b043aff6","Type":"ContainerDied","Data":"1c0cac8c8e93c336b9ce4b9f35ba0531364a4a4da6679b29e397f486330cf060"} Jul 27 21:12:43.840811 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:43.840402 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dcbd5694-ntknm" event={"ID":"ef24c526-4136-46ba-973e-b570b043aff6","Type":"ContainerDied","Data":"7aa0d8e83e2d65dc7985e5c4d72390955b0d7165f771709f264f0fbff350e02f"} Jul 27 21:12:43.840811 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:43.840421 2577 scope.go:117] "RemoveContainer" containerID="1c0cac8c8e93c336b9ce4b9f35ba0531364a4a4da6679b29e397f486330cf060" Jul 27 21:12:43.849471 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:43.849450 2577 scope.go:117] "RemoveContainer" containerID="1c0cac8c8e93c336b9ce4b9f35ba0531364a4a4da6679b29e397f486330cf060" Jul 27 21:12:43.849765 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:12:43.849746 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c0cac8c8e93c336b9ce4b9f35ba0531364a4a4da6679b29e397f486330cf060\": container with ID starting with 1c0cac8c8e93c336b9ce4b9f35ba0531364a4a4da6679b29e397f486330cf060 not found: ID does not exist" containerID="1c0cac8c8e93c336b9ce4b9f35ba0531364a4a4da6679b29e397f486330cf060" Jul 27 21:12:43.849848 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:43.849773 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c0cac8c8e93c336b9ce4b9f35ba0531364a4a4da6679b29e397f486330cf060"} err="failed to get container status \"1c0cac8c8e93c336b9ce4b9f35ba0531364a4a4da6679b29e397f486330cf060\": rpc error: code = NotFound desc = could not find container \"1c0cac8c8e93c336b9ce4b9f35ba0531364a4a4da6679b29e397f486330cf060\": container with ID starting with 1c0cac8c8e93c336b9ce4b9f35ba0531364a4a4da6679b29e397f486330cf060 not found: ID does not exist" Jul 27 21:12:43.860777 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:43.860747 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-dcbd5694-ntknm"] Jul 27 21:12:43.865068 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:43.865042 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-dcbd5694-ntknm"] Jul 27 21:12:44.366664 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:12:44.366632 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef24c526-4136-46ba-973e-b570b043aff6" path="/var/lib/kubelet/pods/ef24c526-4136-46ba-973e-b570b043aff6/volumes" Jul 27 21:13:28.256166 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:13:28.256136 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-j5rmn_4812c628-5f41-4e36-aaf6-8d62e4b6a443/console-operator/2.log" Jul 27 21:13:28.260876 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:13:28.260846 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6mhn_1907dc14-1132-4eb9-a5ee-675b43df6e6d/ovn-acl-logging/0.log" Jul 27 21:13:28.262152 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:13:28.262129 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-j5rmn_4812c628-5f41-4e36-aaf6-8d62e4b6a443/console-operator/2.log" Jul 27 21:13:28.266875 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:13:28.266851 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6mhn_1907dc14-1132-4eb9-a5ee-675b43df6e6d/ovn-acl-logging/0.log" Jul 27 21:16:44.234167 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:16:44.234126 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-b84f4c8c7-h2txg"] Jul 27 21:16:44.234837 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:16:44.234638 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18a95c92-c423-4ce0-878d-f4b4a8d0638a" containerName="manager" Jul 27 21:16:44.234837 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:16:44.234660 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="18a95c92-c423-4ce0-878d-f4b4a8d0638a" containerName="manager" Jul 27 21:16:44.234837 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:16:44.234704 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef24c526-4136-46ba-973e-b570b043aff6" containerName="console" Jul 27 21:16:44.234837 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:16:44.234714 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef24c526-4136-46ba-973e-b570b043aff6" containerName="console" Jul 27 21:16:44.234837 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:16:44.234799 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="18a95c92-c423-4ce0-878d-f4b4a8d0638a" containerName="manager" Jul 27 21:16:44.234837 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:16:44.234814 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef24c526-4136-46ba-973e-b570b043aff6" containerName="console" Jul 27 21:16:44.237994 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:16:44.237968 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-h2txg" Jul 27 21:16:44.240608 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:16:44.240561 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-kube-rbac-proxy-sar-config\"" Jul 27 21:16:44.240608 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:16:44.240594 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 27 21:16:44.240802 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:16:44.240645 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-serving-cert\"" Jul 27 21:16:44.240802 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:16:44.240785 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-wg596\"" Jul 27 21:16:44.245915 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:16:44.245745 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-b84f4c8c7-h2txg"] Jul 27 21:16:44.310746 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:16:44.310709 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7aa50efa-14f9-4085-8781-43dd0901fc6a-openshift-service-ca-bundle\") pod \"model-chainer-b84f4c8c7-h2txg\" (UID: \"7aa50efa-14f9-4085-8781-43dd0901fc6a\") " pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-h2txg" Jul 27 21:16:44.310746 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:16:44.310749 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7aa50efa-14f9-4085-8781-43dd0901fc6a-proxy-tls\") pod \"model-chainer-b84f4c8c7-h2txg\" (UID: \"7aa50efa-14f9-4085-8781-43dd0901fc6a\") " pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-h2txg" Jul 27 21:16:44.412211 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:16:44.412166 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7aa50efa-14f9-4085-8781-43dd0901fc6a-openshift-service-ca-bundle\") pod \"model-chainer-b84f4c8c7-h2txg\" (UID: \"7aa50efa-14f9-4085-8781-43dd0901fc6a\") " pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-h2txg" Jul 27 21:16:44.412211 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:16:44.412226 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7aa50efa-14f9-4085-8781-43dd0901fc6a-proxy-tls\") pod \"model-chainer-b84f4c8c7-h2txg\" (UID: \"7aa50efa-14f9-4085-8781-43dd0901fc6a\") " pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-h2txg" Jul 27 21:16:44.412991 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:16:44.412964 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7aa50efa-14f9-4085-8781-43dd0901fc6a-openshift-service-ca-bundle\") pod \"model-chainer-b84f4c8c7-h2txg\" (UID: \"7aa50efa-14f9-4085-8781-43dd0901fc6a\") " pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-h2txg" Jul 27 21:16:44.414899 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:16:44.414870 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7aa50efa-14f9-4085-8781-43dd0901fc6a-proxy-tls\") pod \"model-chainer-b84f4c8c7-h2txg\" (UID: \"7aa50efa-14f9-4085-8781-43dd0901fc6a\") " pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-h2txg" Jul 27 21:16:44.550209 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:16:44.550123 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-h2txg" Jul 27 21:16:44.687516 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:16:44.687434 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-b84f4c8c7-h2txg"] Jul 27 21:16:44.690472 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:16:44.690441 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aa50efa_14f9_4085_8781_43dd0901fc6a.slice/crio-4af77af34d2210a33b4e536956bdb120b9376fd067a667436df310e39c4ff88b WatchSource:0}: Error finding container 4af77af34d2210a33b4e536956bdb120b9376fd067a667436df310e39c4ff88b: Status 404 returned error can't find the container with id 4af77af34d2210a33b4e536956bdb120b9376fd067a667436df310e39c4ff88b Jul 27 21:16:44.692920 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:16:44.692897 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 27 21:16:44.893118 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:16:44.893028 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-h2txg" event={"ID":"7aa50efa-14f9-4085-8781-43dd0901fc6a","Type":"ContainerStarted","Data":"4af77af34d2210a33b4e536956bdb120b9376fd067a667436df310e39c4ff88b"} Jul 27 21:16:46.904055 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:16:46.903968 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-h2txg" event={"ID":"7aa50efa-14f9-4085-8781-43dd0901fc6a","Type":"ContainerStarted","Data":"d15613a9faf350a440e40adfe2979fdda3b4545d61c633d4b441ad86d7a40f97"} Jul 27 21:16:46.904406 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:16:46.904104 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-h2txg" Jul 27 21:16:46.921214 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:16:46.921163 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-h2txg" podStartSLOduration=1.00497754 podStartE2EDuration="2.921147182s" totalImagesPullingTime="1.916169642s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:16:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:16:46.920344128 +0000 UTC m=+799.051960356" watchObservedRunningTime="2026-07-27 21:16:46.921147182 +0000 UTC m=+799.052763411" Jul 27 21:16:52.914923 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:16:52.914850 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-h2txg" Jul 27 21:16:54.341775 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:16:54.341738 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-b84f4c8c7-h2txg"] Jul 27 21:16:54.342176 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:16:54.341953 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-h2txg" podUID="7aa50efa-14f9-4085-8781-43dd0901fc6a" containerName="model-chainer" containerID="cri-o://d15613a9faf350a440e40adfe2979fdda3b4545d61c633d4b441ad86d7a40f97" gracePeriod=30 Jul 27 21:16:57.913586 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:16:57.913515 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-h2txg" podUID="7aa50efa-14f9-4085-8781-43dd0901fc6a" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 21:17:02.913491 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:17:02.913447 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-h2txg" podUID="7aa50efa-14f9-4085-8781-43dd0901fc6a" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 21:17:07.913647 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:17:07.913610 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-h2txg" podUID="7aa50efa-14f9-4085-8781-43dd0901fc6a" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 21:17:07.914048 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:17:07.913722 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-h2txg" Jul 27 21:17:12.913563 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:17:12.913521 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-h2txg" podUID="7aa50efa-14f9-4085-8781-43dd0901fc6a" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 21:17:17.913477 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:17:17.913428 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-h2txg" podUID="7aa50efa-14f9-4085-8781-43dd0901fc6a" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 21:17:22.913415 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:17:22.913365 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-h2txg" podUID="7aa50efa-14f9-4085-8781-43dd0901fc6a" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 21:17:24.501994 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:17:24.501970 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-h2txg" Jul 27 21:17:24.563703 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:17:24.563663 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7aa50efa-14f9-4085-8781-43dd0901fc6a-openshift-service-ca-bundle\") pod \"7aa50efa-14f9-4085-8781-43dd0901fc6a\" (UID: \"7aa50efa-14f9-4085-8781-43dd0901fc6a\") " Jul 27 21:17:24.563876 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:17:24.563747 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7aa50efa-14f9-4085-8781-43dd0901fc6a-proxy-tls\") pod \"7aa50efa-14f9-4085-8781-43dd0901fc6a\" (UID: \"7aa50efa-14f9-4085-8781-43dd0901fc6a\") " Jul 27 21:17:24.564090 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:17:24.564069 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aa50efa-14f9-4085-8781-43dd0901fc6a-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "7aa50efa-14f9-4085-8781-43dd0901fc6a" (UID: "7aa50efa-14f9-4085-8781-43dd0901fc6a"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 21:17:24.565929 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:17:24.565897 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7aa50efa-14f9-4085-8781-43dd0901fc6a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7aa50efa-14f9-4085-8781-43dd0901fc6a" (UID: "7aa50efa-14f9-4085-8781-43dd0901fc6a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 21:17:24.664410 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:17:24.664337 2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7aa50efa-14f9-4085-8781-43dd0901fc6a-proxy-tls\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\"" Jul 27 21:17:24.664410 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:17:24.664365 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7aa50efa-14f9-4085-8781-43dd0901fc6a-openshift-service-ca-bundle\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\"" Jul 27 21:17:24.726406 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:17:24.726367 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-d355e-8455db45bc-6rwmn"] Jul 27 21:17:24.726727 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:17:24.726714 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7aa50efa-14f9-4085-8781-43dd0901fc6a" containerName="model-chainer" Jul 27 21:17:24.726782 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:17:24.726728 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7aa50efa-14f9-4085-8781-43dd0901fc6a" containerName="model-chainer" Jul 27 21:17:24.726819 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:17:24.726792 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7aa50efa-14f9-4085-8781-43dd0901fc6a" containerName="model-chainer" Jul 27 21:17:24.729793 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:17:24.729775 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-d355e-8455db45bc-6rwmn" Jul 27 21:17:24.732377 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:17:24.732352 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-d355e-kube-rbac-proxy-sar-config\"" Jul 27 21:17:24.732565 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:17:24.732535 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-d355e-serving-cert\"" Jul 27 21:17:24.737847 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:17:24.737825 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-d355e-8455db45bc-6rwmn"] Jul 27 21:17:24.765698 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:17:24.765664 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/daf8ac4e-3698-4f27-a5b3-3cb8247d05c9-proxy-tls\") pod \"switch-graph-d355e-8455db45bc-6rwmn\" (UID: \"daf8ac4e-3698-4f27-a5b3-3cb8247d05c9\") " pod="kserve-ci-e2e-test/switch-graph-d355e-8455db45bc-6rwmn" Jul 27 21:17:24.765867 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:17:24.765735 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/daf8ac4e-3698-4f27-a5b3-3cb8247d05c9-openshift-service-ca-bundle\") pod \"switch-graph-d355e-8455db45bc-6rwmn\" (UID: \"daf8ac4e-3698-4f27-a5b3-3cb8247d05c9\") " pod="kserve-ci-e2e-test/switch-graph-d355e-8455db45bc-6rwmn" Jul 27 21:17:24.866656 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:17:24.866619 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/daf8ac4e-3698-4f27-a5b3-3cb8247d05c9-openshift-service-ca-bundle\") pod \"switch-graph-d355e-8455db45bc-6rwmn\" (UID: \"daf8ac4e-3698-4f27-a5b3-3cb8247d05c9\") " pod="kserve-ci-e2e-test/switch-graph-d355e-8455db45bc-6rwmn" Jul 27 21:17:24.866822 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:17:24.866680 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/daf8ac4e-3698-4f27-a5b3-3cb8247d05c9-proxy-tls\") pod \"switch-graph-d355e-8455db45bc-6rwmn\" (UID: \"daf8ac4e-3698-4f27-a5b3-3cb8247d05c9\") " pod="kserve-ci-e2e-test/switch-graph-d355e-8455db45bc-6rwmn" Jul 27 21:17:24.866822 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:17:24.866794 2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-d355e-serving-cert: secret "switch-graph-d355e-serving-cert" not found Jul 27 21:17:24.866897 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:17:24.866860 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/daf8ac4e-3698-4f27-a5b3-3cb8247d05c9-proxy-tls podName:daf8ac4e-3698-4f27-a5b3-3cb8247d05c9 nodeName:}" failed. No retries permitted until 2026-07-27 21:17:25.366843032 +0000 UTC m=+837.498459232 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/daf8ac4e-3698-4f27-a5b3-3cb8247d05c9-proxy-tls") pod "switch-graph-d355e-8455db45bc-6rwmn" (UID: "daf8ac4e-3698-4f27-a5b3-3cb8247d05c9") : secret "switch-graph-d355e-serving-cert" not found Jul 27 21:17:24.867294 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:17:24.867274 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/daf8ac4e-3698-4f27-a5b3-3cb8247d05c9-openshift-service-ca-bundle\") pod \"switch-graph-d355e-8455db45bc-6rwmn\" (UID: \"daf8ac4e-3698-4f27-a5b3-3cb8247d05c9\") " pod="kserve-ci-e2e-test/switch-graph-d355e-8455db45bc-6rwmn" Jul 27 21:17:25.082760 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:17:25.082724 2577 generic.go:356] "Generic (PLEG): container finished" podID="7aa50efa-14f9-4085-8781-43dd0901fc6a" containerID="d15613a9faf350a440e40adfe2979fdda3b4545d61c633d4b441ad86d7a40f97" exitCode=0 Jul 27 21:17:25.082941 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:17:25.082774 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-h2txg" Jul 27 21:17:25.082941 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:17:25.082811 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-h2txg" event={"ID":"7aa50efa-14f9-4085-8781-43dd0901fc6a","Type":"ContainerDied","Data":"d15613a9faf350a440e40adfe2979fdda3b4545d61c633d4b441ad86d7a40f97"} Jul 27 21:17:25.082941 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:17:25.082868 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-h2txg" event={"ID":"7aa50efa-14f9-4085-8781-43dd0901fc6a","Type":"ContainerDied","Data":"4af77af34d2210a33b4e536956bdb120b9376fd067a667436df310e39c4ff88b"} Jul 27 21:17:25.082941 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:17:25.082891 2577 scope.go:117] "RemoveContainer" containerID="d15613a9faf350a440e40adfe2979fdda3b4545d61c633d4b441ad86d7a40f97" Jul 27 21:17:25.092106 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:17:25.092082 2577 scope.go:117] "RemoveContainer" containerID="d15613a9faf350a440e40adfe2979fdda3b4545d61c633d4b441ad86d7a40f97" Jul 27 21:17:25.092417 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:17:25.092399 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d15613a9faf350a440e40adfe2979fdda3b4545d61c633d4b441ad86d7a40f97\": container with ID starting with d15613a9faf350a440e40adfe2979fdda3b4545d61c633d4b441ad86d7a40f97 not found: ID does not exist" containerID="d15613a9faf350a440e40adfe2979fdda3b4545d61c633d4b441ad86d7a40f97" Jul 27 21:17:25.092473 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:17:25.092427 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d15613a9faf350a440e40adfe2979fdda3b4545d61c633d4b441ad86d7a40f97"} err="failed to get container status \"d15613a9faf350a440e40adfe2979fdda3b4545d61c633d4b441ad86d7a40f97\": rpc error: code = NotFound desc = could not find container \"d15613a9faf350a440e40adfe2979fdda3b4545d61c633d4b441ad86d7a40f97\": container with ID starting with d15613a9faf350a440e40adfe2979fdda3b4545d61c633d4b441ad86d7a40f97 not found: ID does not exist" Jul 27 21:17:25.104680 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:17:25.104645 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-b84f4c8c7-h2txg"] Jul 27 21:17:25.108443 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:17:25.108419 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-b84f4c8c7-h2txg"] Jul 27 21:17:25.371842 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:17:25.371738 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/daf8ac4e-3698-4f27-a5b3-3cb8247d05c9-proxy-tls\") pod \"switch-graph-d355e-8455db45bc-6rwmn\" (UID: \"daf8ac4e-3698-4f27-a5b3-3cb8247d05c9\") " pod="kserve-ci-e2e-test/switch-graph-d355e-8455db45bc-6rwmn" Jul 27 21:17:25.374333 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:17:25.374306 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/daf8ac4e-3698-4f27-a5b3-3cb8247d05c9-proxy-tls\") pod \"switch-graph-d355e-8455db45bc-6rwmn\" (UID: \"daf8ac4e-3698-4f27-a5b3-3cb8247d05c9\") " pod="kserve-ci-e2e-test/switch-graph-d355e-8455db45bc-6rwmn" Jul 27 21:17:25.640719 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:17:25.640623 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-d355e-8455db45bc-6rwmn" Jul 27 21:17:25.776764 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:17:25.776729 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-d355e-8455db45bc-6rwmn"] Jul 27 21:17:25.780018 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:17:25.779993 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaf8ac4e_3698_4f27_a5b3_3cb8247d05c9.slice/crio-5a50e69b09fccdd69f8d9e357ded1e91bb04a30ae71869924b122d9d56995ed9 WatchSource:0}: Error finding container 5a50e69b09fccdd69f8d9e357ded1e91bb04a30ae71869924b122d9d56995ed9: Status 404 returned error can't find the container with id 5a50e69b09fccdd69f8d9e357ded1e91bb04a30ae71869924b122d9d56995ed9 Jul 27 21:17:26.088875 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:17:26.088839 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-d355e-8455db45bc-6rwmn" event={"ID":"daf8ac4e-3698-4f27-a5b3-3cb8247d05c9","Type":"ContainerStarted","Data":"7d8e1e17985ed36a4470d36a0cf4924d7d667249f2308bdaac9cedb8922c9ea3"} Jul 27 21:17:26.089077 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:17:26.088888 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-d355e-8455db45bc-6rwmn" event={"ID":"daf8ac4e-3698-4f27-a5b3-3cb8247d05c9","Type":"ContainerStarted","Data":"5a50e69b09fccdd69f8d9e357ded1e91bb04a30ae71869924b122d9d56995ed9"} Jul 27 21:17:26.089077 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:17:26.088978 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-d355e-8455db45bc-6rwmn" Jul 27 21:17:26.106757 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:17:26.106676 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-d355e-8455db45bc-6rwmn" podStartSLOduration=2.10665869 podStartE2EDuration="2.10665869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:17:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:17:26.106000208 +0000 UTC m=+838.237616430" watchObservedRunningTime="2026-07-27 21:17:26.10665869 +0000 UTC m=+838.238274912" Jul 27 21:17:26.372279 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:17:26.372182 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7aa50efa-14f9-4085-8781-43dd0901fc6a" path="/var/lib/kubelet/pods/7aa50efa-14f9-4085-8781-43dd0901fc6a/volumes" Jul 27 21:17:32.100880 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:17:32.100848 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-d355e-8455db45bc-6rwmn" Jul 27 21:17:54.541301 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:17:54.541265 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-d13b2-8dc58d59b-hjbgw"] Jul 27 21:17:54.544681 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:17:54.544663 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-d13b2-8dc58d59b-hjbgw" Jul 27 21:17:54.547274 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:17:54.547247 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-d13b2-serving-cert\"" Jul 27 21:17:54.547405 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:17:54.547332 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-d13b2-kube-rbac-proxy-sar-config\"" Jul 27 21:17:54.555932 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:17:54.555905 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-d13b2-8dc58d59b-hjbgw"] Jul 27 21:17:54.625960 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:17:54.625928 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/717291e3-5be8-4acd-ba06-17079b208344-openshift-service-ca-bundle\") pod \"sequence-graph-d13b2-8dc58d59b-hjbgw\" (UID: \"717291e3-5be8-4acd-ba06-17079b208344\") " pod="kserve-ci-e2e-test/sequence-graph-d13b2-8dc58d59b-hjbgw" Jul 27 21:17:54.626122 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:17:54.625979 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/717291e3-5be8-4acd-ba06-17079b208344-proxy-tls\") pod \"sequence-graph-d13b2-8dc58d59b-hjbgw\" (UID: \"717291e3-5be8-4acd-ba06-17079b208344\") " pod="kserve-ci-e2e-test/sequence-graph-d13b2-8dc58d59b-hjbgw" Jul 27 21:17:54.726958 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:17:54.726920 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/717291e3-5be8-4acd-ba06-17079b208344-openshift-service-ca-bundle\") pod \"sequence-graph-d13b2-8dc58d59b-hjbgw\" (UID: \"717291e3-5be8-4acd-ba06-17079b208344\") " pod="kserve-ci-e2e-test/sequence-graph-d13b2-8dc58d59b-hjbgw" Jul 27 21:17:54.727131 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:17:54.726967 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/717291e3-5be8-4acd-ba06-17079b208344-proxy-tls\") pod \"sequence-graph-d13b2-8dc58d59b-hjbgw\" (UID: \"717291e3-5be8-4acd-ba06-17079b208344\") " pod="kserve-ci-e2e-test/sequence-graph-d13b2-8dc58d59b-hjbgw" Jul 27 21:17:54.727131 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:17:54.727068 2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-d13b2-serving-cert: secret "sequence-graph-d13b2-serving-cert" not found Jul 27 21:17:54.727131 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:17:54.727124 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/717291e3-5be8-4acd-ba06-17079b208344-proxy-tls podName:717291e3-5be8-4acd-ba06-17079b208344 nodeName:}" failed. No retries permitted until 2026-07-27 21:17:55.227108779 +0000 UTC m=+867.358724979 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/717291e3-5be8-4acd-ba06-17079b208344-proxy-tls") pod "sequence-graph-d13b2-8dc58d59b-hjbgw" (UID: "717291e3-5be8-4acd-ba06-17079b208344") : secret "sequence-graph-d13b2-serving-cert" not found Jul 27 21:17:54.727668 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:17:54.727647 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/717291e3-5be8-4acd-ba06-17079b208344-openshift-service-ca-bundle\") pod \"sequence-graph-d13b2-8dc58d59b-hjbgw\" (UID: \"717291e3-5be8-4acd-ba06-17079b208344\") " pod="kserve-ci-e2e-test/sequence-graph-d13b2-8dc58d59b-hjbgw" Jul 27 21:17:55.230246 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:17:55.230205 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/717291e3-5be8-4acd-ba06-17079b208344-proxy-tls\") pod \"sequence-graph-d13b2-8dc58d59b-hjbgw\" (UID: \"717291e3-5be8-4acd-ba06-17079b208344\") " pod="kserve-ci-e2e-test/sequence-graph-d13b2-8dc58d59b-hjbgw" Jul 27 21:17:55.232620 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:17:55.232598 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/717291e3-5be8-4acd-ba06-17079b208344-proxy-tls\") pod \"sequence-graph-d13b2-8dc58d59b-hjbgw\" (UID: \"717291e3-5be8-4acd-ba06-17079b208344\") " pod="kserve-ci-e2e-test/sequence-graph-d13b2-8dc58d59b-hjbgw" Jul 27 21:17:55.455612 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:17:55.455560 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-d13b2-8dc58d59b-hjbgw" Jul 27 21:17:55.607880 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:17:55.607850 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-d13b2-8dc58d59b-hjbgw"] Jul 27 21:17:55.610630 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:17:55.610598 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod717291e3_5be8_4acd_ba06_17079b208344.slice/crio-e8b46561fe7562e53fe5781a38b4840bd8b790a45ece12fbd09434bc08e62b7c WatchSource:0}: Error finding container e8b46561fe7562e53fe5781a38b4840bd8b790a45ece12fbd09434bc08e62b7c: Status 404 returned error can't find the container with id e8b46561fe7562e53fe5781a38b4840bd8b790a45ece12fbd09434bc08e62b7c Jul 27 21:17:56.228125 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:17:56.228072 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-d13b2-8dc58d59b-hjbgw" event={"ID":"717291e3-5be8-4acd-ba06-17079b208344","Type":"ContainerStarted","Data":"bbac27a96999f640ff6d3aee65d482ea41e441fd4f8a173a52eb7ded71d409aa"} Jul 27 21:17:56.228320 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:17:56.228141 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-d13b2-8dc58d59b-hjbgw" event={"ID":"717291e3-5be8-4acd-ba06-17079b208344","Type":"ContainerStarted","Data":"e8b46561fe7562e53fe5781a38b4840bd8b790a45ece12fbd09434bc08e62b7c"} Jul 27 21:17:56.228320 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:17:56.228173 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-d13b2-8dc58d59b-hjbgw" Jul 27 21:17:56.247595 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:17:56.247516 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-d13b2-8dc58d59b-hjbgw" podStartSLOduration=2.247499617 podStartE2EDuration="2.247499617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:17:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:17:56.246067252 +0000 UTC m=+868.377683475" watchObservedRunningTime="2026-07-27 21:17:56.247499617 +0000 UTC m=+868.379115839" Jul 27 21:18:02.238904 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:18:02.238870 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-d13b2-8dc58d59b-hjbgw" Jul 27 21:18:28.292053 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:18:28.291968 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-j5rmn_4812c628-5f41-4e36-aaf6-8d62e4b6a443/console-operator/2.log" Jul 27 21:18:28.294072 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:18:28.294043 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-j5rmn_4812c628-5f41-4e36-aaf6-8d62e4b6a443/console-operator/2.log" Jul 27 21:18:28.296766 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:18:28.296746 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6mhn_1907dc14-1132-4eb9-a5ee-675b43df6e6d/ovn-acl-logging/0.log" Jul 27 21:18:28.298635 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:18:28.298615 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6mhn_1907dc14-1132-4eb9-a5ee-675b43df6e6d/ovn-acl-logging/0.log" Jul 27 21:23:28.320950 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:23:28.320870 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-j5rmn_4812c628-5f41-4e36-aaf6-8d62e4b6a443/console-operator/2.log" Jul 27 21:23:28.321494 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:23:28.320947 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-j5rmn_4812c628-5f41-4e36-aaf6-8d62e4b6a443/console-operator/2.log" Jul 27 21:23:28.325691 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:23:28.325661 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6mhn_1907dc14-1132-4eb9-a5ee-675b43df6e6d/ovn-acl-logging/0.log" Jul 27 21:23:28.325947 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:23:28.325928 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6mhn_1907dc14-1132-4eb9-a5ee-675b43df6e6d/ovn-acl-logging/0.log" Jul 27 21:25:39.332456 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:25:39.332422 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-d355e-8455db45bc-6rwmn"] Jul 27 21:25:39.333009 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:25:39.332622 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-d355e-8455db45bc-6rwmn" podUID="daf8ac4e-3698-4f27-a5b3-3cb8247d05c9" containerName="switch-graph-d355e" containerID="cri-o://7d8e1e17985ed36a4470d36a0cf4924d7d667249f2308bdaac9cedb8922c9ea3" gracePeriod=30 Jul 27 21:25:42.099025 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:25:42.098984 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-d355e-8455db45bc-6rwmn" podUID="daf8ac4e-3698-4f27-a5b3-3cb8247d05c9" containerName="switch-graph-d355e" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 21:25:47.098822 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:25:47.098774 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-d355e-8455db45bc-6rwmn" podUID="daf8ac4e-3698-4f27-a5b3-3cb8247d05c9" containerName="switch-graph-d355e" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 21:25:52.098474 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:25:52.098377 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-d355e-8455db45bc-6rwmn" podUID="daf8ac4e-3698-4f27-a5b3-3cb8247d05c9" containerName="switch-graph-d355e" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 21:25:52.098897 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:25:52.098560 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-d355e-8455db45bc-6rwmn" Jul 27 21:25:57.098858 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:25:57.098817 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-d355e-8455db45bc-6rwmn" podUID="daf8ac4e-3698-4f27-a5b3-3cb8247d05c9" containerName="switch-graph-d355e" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 21:26:02.098506 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:02.098467 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-d355e-8455db45bc-6rwmn" podUID="daf8ac4e-3698-4f27-a5b3-3cb8247d05c9" containerName="switch-graph-d355e" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 21:26:07.098659 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:07.098612 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-d355e-8455db45bc-6rwmn" podUID="daf8ac4e-3698-4f27-a5b3-3cb8247d05c9" containerName="switch-graph-d355e" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 21:26:09.131640 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:09.131604 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-d13b2-8dc58d59b-hjbgw"] Jul 27 21:26:09.132117 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:09.131872 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-d13b2-8dc58d59b-hjbgw" podUID="717291e3-5be8-4acd-ba06-17079b208344" containerName="sequence-graph-d13b2" containerID="cri-o://bbac27a96999f640ff6d3aee65d482ea41e441fd4f8a173a52eb7ded71d409aa" gracePeriod=30 Jul 27 21:26:09.472377 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:09.472335 2577 generic.go:356] "Generic (PLEG): container finished" podID="daf8ac4e-3698-4f27-a5b3-3cb8247d05c9" containerID="7d8e1e17985ed36a4470d36a0cf4924d7d667249f2308bdaac9cedb8922c9ea3" exitCode=0 Jul 27 21:26:09.472583 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:09.472398 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-d355e-8455db45bc-6rwmn" event={"ID":"daf8ac4e-3698-4f27-a5b3-3cb8247d05c9","Type":"ContainerDied","Data":"7d8e1e17985ed36a4470d36a0cf4924d7d667249f2308bdaac9cedb8922c9ea3"} Jul 27 21:26:09.505404 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:09.505376 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-d355e-8455db45bc-6rwmn" Jul 27 21:26:09.508799 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:09.508772 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/daf8ac4e-3698-4f27-a5b3-3cb8247d05c9-proxy-tls\") pod \"daf8ac4e-3698-4f27-a5b3-3cb8247d05c9\" (UID: \"daf8ac4e-3698-4f27-a5b3-3cb8247d05c9\") " Jul 27 21:26:09.508946 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:09.508834 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/daf8ac4e-3698-4f27-a5b3-3cb8247d05c9-openshift-service-ca-bundle\") pod \"daf8ac4e-3698-4f27-a5b3-3cb8247d05c9\" (UID: \"daf8ac4e-3698-4f27-a5b3-3cb8247d05c9\") " Jul 27 21:26:09.509473 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:09.509439 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daf8ac4e-3698-4f27-a5b3-3cb8247d05c9-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "daf8ac4e-3698-4f27-a5b3-3cb8247d05c9" (UID: "daf8ac4e-3698-4f27-a5b3-3cb8247d05c9"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 21:26:09.511041 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:09.511014 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daf8ac4e-3698-4f27-a5b3-3cb8247d05c9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "daf8ac4e-3698-4f27-a5b3-3cb8247d05c9" (UID: "daf8ac4e-3698-4f27-a5b3-3cb8247d05c9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 21:26:09.610028 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:09.609991 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/daf8ac4e-3698-4f27-a5b3-3cb8247d05c9-openshift-service-ca-bundle\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\"" Jul 27 21:26:09.610028 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:09.610024 2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/daf8ac4e-3698-4f27-a5b3-3cb8247d05c9-proxy-tls\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\"" Jul 27 21:26:10.478739 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:10.478712 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-d355e-8455db45bc-6rwmn" Jul 27 21:26:10.478739 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:10.478736 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-d355e-8455db45bc-6rwmn" event={"ID":"daf8ac4e-3698-4f27-a5b3-3cb8247d05c9","Type":"ContainerDied","Data":"5a50e69b09fccdd69f8d9e357ded1e91bb04a30ae71869924b122d9d56995ed9"} Jul 27 21:26:10.479348 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:10.478798 2577 scope.go:117] "RemoveContainer" containerID="7d8e1e17985ed36a4470d36a0cf4924d7d667249f2308bdaac9cedb8922c9ea3" Jul 27 21:26:10.495355 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:10.495323 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-d355e-8455db45bc-6rwmn"] Jul 27 21:26:10.501072 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:10.501042 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-d355e-8455db45bc-6rwmn"] Jul 27 21:26:12.237766 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:12.237721 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-d13b2-8dc58d59b-hjbgw" podUID="717291e3-5be8-4acd-ba06-17079b208344" containerName="sequence-graph-d13b2" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 21:26:12.366999 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:12.366962 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="daf8ac4e-3698-4f27-a5b3-3cb8247d05c9" path="/var/lib/kubelet/pods/daf8ac4e-3698-4f27-a5b3-3cb8247d05c9/volumes" Jul 27 21:26:17.237044 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:17.237000 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-d13b2-8dc58d59b-hjbgw" podUID="717291e3-5be8-4acd-ba06-17079b208344" containerName="sequence-graph-d13b2" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 21:26:22.237698 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:22.237657 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-d13b2-8dc58d59b-hjbgw" podUID="717291e3-5be8-4acd-ba06-17079b208344" containerName="sequence-graph-d13b2" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 21:26:22.238087 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:22.237778 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-d13b2-8dc58d59b-hjbgw" Jul 27 21:26:27.237962 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:27.237921 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-d13b2-8dc58d59b-hjbgw" podUID="717291e3-5be8-4acd-ba06-17079b208344" containerName="sequence-graph-d13b2" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 21:26:32.237771 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:32.237728 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-d13b2-8dc58d59b-hjbgw" podUID="717291e3-5be8-4acd-ba06-17079b208344" containerName="sequence-graph-d13b2" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 21:26:37.237691 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:37.237646 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-d13b2-8dc58d59b-hjbgw" podUID="717291e3-5be8-4acd-ba06-17079b208344" containerName="sequence-graph-d13b2" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 21:26:39.293645 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:39.293616 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-d13b2-8dc58d59b-hjbgw" Jul 27 21:26:39.365263 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:39.365230 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/717291e3-5be8-4acd-ba06-17079b208344-openshift-service-ca-bundle\") pod \"717291e3-5be8-4acd-ba06-17079b208344\" (UID: \"717291e3-5be8-4acd-ba06-17079b208344\") " Jul 27 21:26:39.365469 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:39.365345 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/717291e3-5be8-4acd-ba06-17079b208344-proxy-tls\") pod \"717291e3-5be8-4acd-ba06-17079b208344\" (UID: \"717291e3-5be8-4acd-ba06-17079b208344\") " Jul 27 21:26:39.365812 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:39.365780 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/717291e3-5be8-4acd-ba06-17079b208344-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "717291e3-5be8-4acd-ba06-17079b208344" (UID: "717291e3-5be8-4acd-ba06-17079b208344"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 21:26:39.367532 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:39.367506 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/717291e3-5be8-4acd-ba06-17079b208344-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "717291e3-5be8-4acd-ba06-17079b208344" (UID: "717291e3-5be8-4acd-ba06-17079b208344"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 21:26:39.466863 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:39.466774 2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/717291e3-5be8-4acd-ba06-17079b208344-proxy-tls\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\"" Jul 27 21:26:39.466863 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:39.466806 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/717291e3-5be8-4acd-ba06-17079b208344-openshift-service-ca-bundle\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\"" Jul 27 21:26:39.609095 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:39.609054 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-cb683-6bd9cdd588-hncbs"] Jul 27 21:26:39.609447 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:39.609433 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="717291e3-5be8-4acd-ba06-17079b208344" containerName="sequence-graph-d13b2" Jul 27 21:26:39.609495 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:39.609450 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="717291e3-5be8-4acd-ba06-17079b208344" containerName="sequence-graph-d13b2" Jul 27 21:26:39.609495 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:39.609467 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="daf8ac4e-3698-4f27-a5b3-3cb8247d05c9" containerName="switch-graph-d355e" Jul 27 21:26:39.609495 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:39.609473 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="daf8ac4e-3698-4f27-a5b3-3cb8247d05c9" containerName="switch-graph-d355e" Jul 27 21:26:39.609613 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:39.609532 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="daf8ac4e-3698-4f27-a5b3-3cb8247d05c9" containerName="switch-graph-d355e" Jul 27 21:26:39.609613 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:39.609542 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="717291e3-5be8-4acd-ba06-17079b208344" containerName="sequence-graph-d13b2" Jul 27 21:26:39.612581 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:39.612552 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-cb683-6bd9cdd588-hncbs" Jul 27 21:26:39.614983 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:39.614956 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-cb683-kube-rbac-proxy-sar-config\"" Jul 27 21:26:39.615094 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:39.614985 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-cb683-serving-cert\"" Jul 27 21:26:39.621473 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:39.621446 2577 generic.go:356] "Generic (PLEG): container finished" podID="717291e3-5be8-4acd-ba06-17079b208344" containerID="bbac27a96999f640ff6d3aee65d482ea41e441fd4f8a173a52eb7ded71d409aa" exitCode=0 Jul 27 21:26:39.621607 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:39.621488 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-d13b2-8dc58d59b-hjbgw" event={"ID":"717291e3-5be8-4acd-ba06-17079b208344","Type":"ContainerDied","Data":"bbac27a96999f640ff6d3aee65d482ea41e441fd4f8a173a52eb7ded71d409aa"} Jul 27 21:26:39.621607 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:39.621500 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-d13b2-8dc58d59b-hjbgw" Jul 27 21:26:39.621723 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:39.621603 2577 scope.go:117] "RemoveContainer" containerID="bbac27a96999f640ff6d3aee65d482ea41e441fd4f8a173a52eb7ded71d409aa" Jul 27 21:26:39.621723 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:39.621559 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-d13b2-8dc58d59b-hjbgw" event={"ID":"717291e3-5be8-4acd-ba06-17079b208344","Type":"ContainerDied","Data":"e8b46561fe7562e53fe5781a38b4840bd8b790a45ece12fbd09434bc08e62b7c"} Jul 27 21:26:39.625687 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:39.625664 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-cb683-6bd9cdd588-hncbs"] Jul 27 21:26:39.631586 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:39.631551 2577 scope.go:117] "RemoveContainer" containerID="bbac27a96999f640ff6d3aee65d482ea41e441fd4f8a173a52eb7ded71d409aa" Jul 27 21:26:39.631996 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:26:39.631971 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbac27a96999f640ff6d3aee65d482ea41e441fd4f8a173a52eb7ded71d409aa\": container with ID starting with bbac27a96999f640ff6d3aee65d482ea41e441fd4f8a173a52eb7ded71d409aa not found: ID does not exist" containerID="bbac27a96999f640ff6d3aee65d482ea41e441fd4f8a173a52eb7ded71d409aa" Jul 27 21:26:39.632060 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:39.632009 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbac27a96999f640ff6d3aee65d482ea41e441fd4f8a173a52eb7ded71d409aa"} err="failed to get container status \"bbac27a96999f640ff6d3aee65d482ea41e441fd4f8a173a52eb7ded71d409aa\": rpc error: code = NotFound desc = could not find container \"bbac27a96999f640ff6d3aee65d482ea41e441fd4f8a173a52eb7ded71d409aa\": container with ID starting with bbac27a96999f640ff6d3aee65d482ea41e441fd4f8a173a52eb7ded71d409aa not found: ID does not exist" Jul 27 21:26:39.645971 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:39.645944 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-d13b2-8dc58d59b-hjbgw"] Jul 27 21:26:39.649037 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:39.649016 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-d13b2-8dc58d59b-hjbgw"] Jul 27 21:26:39.769604 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:39.769537 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84913fee-eaea-4b13-b8df-d0c70596ad75-proxy-tls\") pod \"ensemble-graph-cb683-6bd9cdd588-hncbs\" (UID: \"84913fee-eaea-4b13-b8df-d0c70596ad75\") " pod="kserve-ci-e2e-test/ensemble-graph-cb683-6bd9cdd588-hncbs" Jul 27 21:26:39.769799 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:39.769698 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84913fee-eaea-4b13-b8df-d0c70596ad75-openshift-service-ca-bundle\") pod \"ensemble-graph-cb683-6bd9cdd588-hncbs\" (UID: \"84913fee-eaea-4b13-b8df-d0c70596ad75\") " pod="kserve-ci-e2e-test/ensemble-graph-cb683-6bd9cdd588-hncbs" Jul 27 21:26:39.871127 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:39.871084 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84913fee-eaea-4b13-b8df-d0c70596ad75-openshift-service-ca-bundle\") pod \"ensemble-graph-cb683-6bd9cdd588-hncbs\" (UID: \"84913fee-eaea-4b13-b8df-d0c70596ad75\") " pod="kserve-ci-e2e-test/ensemble-graph-cb683-6bd9cdd588-hncbs" Jul 27 21:26:39.871320 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:39.871174 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84913fee-eaea-4b13-b8df-d0c70596ad75-proxy-tls\") pod \"ensemble-graph-cb683-6bd9cdd588-hncbs\" (UID: \"84913fee-eaea-4b13-b8df-d0c70596ad75\") " pod="kserve-ci-e2e-test/ensemble-graph-cb683-6bd9cdd588-hncbs" Jul 27 21:26:39.871800 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:39.871775 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84913fee-eaea-4b13-b8df-d0c70596ad75-openshift-service-ca-bundle\") pod \"ensemble-graph-cb683-6bd9cdd588-hncbs\" (UID: \"84913fee-eaea-4b13-b8df-d0c70596ad75\") " pod="kserve-ci-e2e-test/ensemble-graph-cb683-6bd9cdd588-hncbs" Jul 27 21:26:39.873521 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:39.873500 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84913fee-eaea-4b13-b8df-d0c70596ad75-proxy-tls\") pod \"ensemble-graph-cb683-6bd9cdd588-hncbs\" (UID: \"84913fee-eaea-4b13-b8df-d0c70596ad75\") " pod="kserve-ci-e2e-test/ensemble-graph-cb683-6bd9cdd588-hncbs" Jul 27 21:26:39.924286 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:39.924243 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-cb683-6bd9cdd588-hncbs" Jul 27 21:26:40.059207 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:40.059114 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-cb683-6bd9cdd588-hncbs"] Jul 27 21:26:40.062177 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:26:40.062148 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84913fee_eaea_4b13_b8df_d0c70596ad75.slice/crio-0872e19925ae61b8c331d053d4ad056128b4978ee6818bc0c82fe34d6c8fad81 WatchSource:0}: Error finding container 0872e19925ae61b8c331d053d4ad056128b4978ee6818bc0c82fe34d6c8fad81: Status 404 returned error can't find the container with id 0872e19925ae61b8c331d053d4ad056128b4978ee6818bc0c82fe34d6c8fad81 Jul 27 21:26:40.366880 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:40.366802 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="717291e3-5be8-4acd-ba06-17079b208344" path="/var/lib/kubelet/pods/717291e3-5be8-4acd-ba06-17079b208344/volumes" Jul 27 21:26:40.627264 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:40.627152 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-cb683-6bd9cdd588-hncbs" event={"ID":"84913fee-eaea-4b13-b8df-d0c70596ad75","Type":"ContainerStarted","Data":"54dcf903daff0970e7ac715d23242e527c275e967ddd3dddc1331624a0d32377"} Jul 27 21:26:40.627264 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:40.627216 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-cb683-6bd9cdd588-hncbs" event={"ID":"84913fee-eaea-4b13-b8df-d0c70596ad75","Type":"ContainerStarted","Data":"0872e19925ae61b8c331d053d4ad056128b4978ee6818bc0c82fe34d6c8fad81"} Jul 27 21:26:40.627264 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:40.627246 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-cb683-6bd9cdd588-hncbs" Jul 27 21:26:40.644012 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:40.643961 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-cb683-6bd9cdd588-hncbs" podStartSLOduration=1.6439437749999999 podStartE2EDuration="1.643943775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:26:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:26:40.642733775 +0000 UTC m=+1392.774349996" watchObservedRunningTime="2026-07-27 21:26:40.643943775 +0000 UTC m=+1392.775559998" Jul 27 21:26:46.638088 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:46.638054 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-cb683-6bd9cdd588-hncbs" Jul 27 21:26:49.677159 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:49.677125 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-cb683-6bd9cdd588-hncbs"] Jul 27 21:26:49.677679 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:49.677300 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-cb683-6bd9cdd588-hncbs" podUID="84913fee-eaea-4b13-b8df-d0c70596ad75" containerName="ensemble-graph-cb683" containerID="cri-o://54dcf903daff0970e7ac715d23242e527c275e967ddd3dddc1331624a0d32377" gracePeriod=30 Jul 27 21:26:51.636549 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:51.636504 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-cb683-6bd9cdd588-hncbs" podUID="84913fee-eaea-4b13-b8df-d0c70596ad75" containerName="ensemble-graph-cb683" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 21:26:56.636682 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:26:56.636632 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-cb683-6bd9cdd588-hncbs" podUID="84913fee-eaea-4b13-b8df-d0c70596ad75" containerName="ensemble-graph-cb683" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 21:27:01.637139 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:27:01.637087 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-cb683-6bd9cdd588-hncbs" podUID="84913fee-eaea-4b13-b8df-d0c70596ad75" containerName="ensemble-graph-cb683" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 21:27:01.637553 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:27:01.637235 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-cb683-6bd9cdd588-hncbs" Jul 27 21:27:06.637401 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:27:06.637353 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-cb683-6bd9cdd588-hncbs" podUID="84913fee-eaea-4b13-b8df-d0c70596ad75" containerName="ensemble-graph-cb683" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 21:27:09.315039 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:27:09.315006 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-0a095-6b6b78bdf9-crcwm"] Jul 27 21:27:09.318462 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:27:09.318443 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-0a095-6b6b78bdf9-crcwm" Jul 27 21:27:09.320749 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:27:09.320725 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-0a095-serving-cert\"" Jul 27 21:27:09.320851 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:27:09.320726 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-0a095-kube-rbac-proxy-sar-config\"" Jul 27 21:27:09.326618 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:27:09.326596 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-0a095-6b6b78bdf9-crcwm"] Jul 27 21:27:09.435623 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:27:09.435564 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa6fb367-4009-42ea-b1b7-0d36f1f2aa0a-proxy-tls\") pod \"sequence-graph-0a095-6b6b78bdf9-crcwm\" (UID: \"aa6fb367-4009-42ea-b1b7-0d36f1f2aa0a\") " pod="kserve-ci-e2e-test/sequence-graph-0a095-6b6b78bdf9-crcwm" Jul 27 21:27:09.435850 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:27:09.435701 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa6fb367-4009-42ea-b1b7-0d36f1f2aa0a-openshift-service-ca-bundle\") pod \"sequence-graph-0a095-6b6b78bdf9-crcwm\" (UID: \"aa6fb367-4009-42ea-b1b7-0d36f1f2aa0a\") " pod="kserve-ci-e2e-test/sequence-graph-0a095-6b6b78bdf9-crcwm" Jul 27 21:27:09.536153 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:27:09.536116 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa6fb367-4009-42ea-b1b7-0d36f1f2aa0a-proxy-tls\") pod \"sequence-graph-0a095-6b6b78bdf9-crcwm\" (UID: \"aa6fb367-4009-42ea-b1b7-0d36f1f2aa0a\") " pod="kserve-ci-e2e-test/sequence-graph-0a095-6b6b78bdf9-crcwm" Jul 27 21:27:09.536349 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:27:09.536165 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa6fb367-4009-42ea-b1b7-0d36f1f2aa0a-openshift-service-ca-bundle\") pod \"sequence-graph-0a095-6b6b78bdf9-crcwm\" (UID: \"aa6fb367-4009-42ea-b1b7-0d36f1f2aa0a\") " pod="kserve-ci-e2e-test/sequence-graph-0a095-6b6b78bdf9-crcwm" Jul 27 21:27:09.536804 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:27:09.536785 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa6fb367-4009-42ea-b1b7-0d36f1f2aa0a-openshift-service-ca-bundle\") pod \"sequence-graph-0a095-6b6b78bdf9-crcwm\" (UID: \"aa6fb367-4009-42ea-b1b7-0d36f1f2aa0a\") " pod="kserve-ci-e2e-test/sequence-graph-0a095-6b6b78bdf9-crcwm" Jul 27 21:27:09.538735 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:27:09.538705 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa6fb367-4009-42ea-b1b7-0d36f1f2aa0a-proxy-tls\") pod \"sequence-graph-0a095-6b6b78bdf9-crcwm\" (UID: \"aa6fb367-4009-42ea-b1b7-0d36f1f2aa0a\") " pod="kserve-ci-e2e-test/sequence-graph-0a095-6b6b78bdf9-crcwm" Jul 27 21:27:09.630115 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:27:09.630010 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-0a095-6b6b78bdf9-crcwm" Jul 27 21:27:09.763580 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:27:09.763534 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-0a095-6b6b78bdf9-crcwm"] Jul 27 21:27:09.767062 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:27:09.767035 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa6fb367_4009_42ea_b1b7_0d36f1f2aa0a.slice/crio-fbed982da7e3a10f8133fa8aa18a20b6be1313ad104051b3d003368b65ed8631 WatchSource:0}: Error finding container fbed982da7e3a10f8133fa8aa18a20b6be1313ad104051b3d003368b65ed8631: Status 404 returned error can't find the container with id fbed982da7e3a10f8133fa8aa18a20b6be1313ad104051b3d003368b65ed8631 Jul 27 21:27:10.762428 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:27:10.762390 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-0a095-6b6b78bdf9-crcwm" event={"ID":"aa6fb367-4009-42ea-b1b7-0d36f1f2aa0a","Type":"ContainerStarted","Data":"811bb70ea3034bfeeceee36653877343ec05381fdf812719a0387aa61ba8d168"} Jul 27 21:27:10.762872 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:27:10.762446 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-0a095-6b6b78bdf9-crcwm" event={"ID":"aa6fb367-4009-42ea-b1b7-0d36f1f2aa0a","Type":"ContainerStarted","Data":"fbed982da7e3a10f8133fa8aa18a20b6be1313ad104051b3d003368b65ed8631"} Jul 27 21:27:10.762872 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:27:10.762471 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-0a095-6b6b78bdf9-crcwm" Jul 27 21:27:10.779839 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:27:10.779787 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-0a095-6b6b78bdf9-crcwm" podStartSLOduration=1.779768435 podStartE2EDuration="1.779768435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:27:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:27:10.77944687 +0000 UTC m=+1422.911063094" watchObservedRunningTime="2026-07-27 21:27:10.779768435 +0000 UTC m=+1422.911384658" Jul 27 21:27:11.636768 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:27:11.636720 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-cb683-6bd9cdd588-hncbs" podUID="84913fee-eaea-4b13-b8df-d0c70596ad75" containerName="ensemble-graph-cb683" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 21:27:16.637527 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:27:16.637486 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-cb683-6bd9cdd588-hncbs" podUID="84913fee-eaea-4b13-b8df-d0c70596ad75" containerName="ensemble-graph-cb683" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 21:27:16.772084 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:27:16.772054 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-0a095-6b6b78bdf9-crcwm" Jul 27 21:27:19.397993 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:27:19.397957 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-0a095-6b6b78bdf9-crcwm"] Jul 27 21:27:19.398402 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:27:19.398150 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-0a095-6b6b78bdf9-crcwm" podUID="aa6fb367-4009-42ea-b1b7-0d36f1f2aa0a" containerName="sequence-graph-0a095" containerID="cri-o://811bb70ea3034bfeeceee36653877343ec05381fdf812719a0387aa61ba8d168" gracePeriod=30 Jul 27 21:27:19.805058 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:27:19.805021 2577 generic.go:356] "Generic (PLEG): container finished" podID="84913fee-eaea-4b13-b8df-d0c70596ad75" containerID="54dcf903daff0970e7ac715d23242e527c275e967ddd3dddc1331624a0d32377" exitCode=0 Jul 27 21:27:19.805205 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:27:19.805104 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-cb683-6bd9cdd588-hncbs" event={"ID":"84913fee-eaea-4b13-b8df-d0c70596ad75","Type":"ContainerDied","Data":"54dcf903daff0970e7ac715d23242e527c275e967ddd3dddc1331624a0d32377"} Jul 27 21:27:19.833700 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:27:19.833674 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-cb683-6bd9cdd588-hncbs" Jul 27 21:27:19.933284 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:27:19.933177 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84913fee-eaea-4b13-b8df-d0c70596ad75-openshift-service-ca-bundle\") pod \"84913fee-eaea-4b13-b8df-d0c70596ad75\" (UID: \"84913fee-eaea-4b13-b8df-d0c70596ad75\") " Jul 27 21:27:19.933284 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:27:19.933283 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84913fee-eaea-4b13-b8df-d0c70596ad75-proxy-tls\") pod \"84913fee-eaea-4b13-b8df-d0c70596ad75\" (UID: \"84913fee-eaea-4b13-b8df-d0c70596ad75\") " Jul 27 21:27:19.933651 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:27:19.933625 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84913fee-eaea-4b13-b8df-d0c70596ad75-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "84913fee-eaea-4b13-b8df-d0c70596ad75" (UID: "84913fee-eaea-4b13-b8df-d0c70596ad75"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 21:27:19.935395 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:27:19.935373 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84913fee-eaea-4b13-b8df-d0c70596ad75-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "84913fee-eaea-4b13-b8df-d0c70596ad75" (UID: "84913fee-eaea-4b13-b8df-d0c70596ad75"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 21:27:20.034786 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:27:20.034747 2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84913fee-eaea-4b13-b8df-d0c70596ad75-proxy-tls\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\"" Jul 27 21:27:20.034786 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:27:20.034780 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84913fee-eaea-4b13-b8df-d0c70596ad75-openshift-service-ca-bundle\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\"" Jul 27 21:27:20.810382 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:27:20.810357 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-cb683-6bd9cdd588-hncbs" Jul 27 21:27:20.810382 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:27:20.810374 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-cb683-6bd9cdd588-hncbs" event={"ID":"84913fee-eaea-4b13-b8df-d0c70596ad75","Type":"ContainerDied","Data":"0872e19925ae61b8c331d053d4ad056128b4978ee6818bc0c82fe34d6c8fad81"} Jul 27 21:27:20.810898 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:27:20.810426 2577 scope.go:117] "RemoveContainer" containerID="54dcf903daff0970e7ac715d23242e527c275e967ddd3dddc1331624a0d32377" Jul 27 21:27:20.826426 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:27:20.826397 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-cb683-6bd9cdd588-hncbs"] Jul 27 21:27:20.832182 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:27:20.832152 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-cb683-6bd9cdd588-hncbs"] Jul 27 21:27:21.770843 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:27:21.770796 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-0a095-6b6b78bdf9-crcwm" podUID="aa6fb367-4009-42ea-b1b7-0d36f1f2aa0a" containerName="sequence-graph-0a095" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 21:27:22.366244 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:27:22.366209 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84913fee-eaea-4b13-b8df-d0c70596ad75" path="/var/lib/kubelet/pods/84913fee-eaea-4b13-b8df-d0c70596ad75/volumes" Jul 27 21:27:26.771249 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:27:26.771201 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-0a095-6b6b78bdf9-crcwm" podUID="aa6fb367-4009-42ea-b1b7-0d36f1f2aa0a" containerName="sequence-graph-0a095" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 21:27:31.770996 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:27:31.770946 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-0a095-6b6b78bdf9-crcwm" podUID="aa6fb367-4009-42ea-b1b7-0d36f1f2aa0a" containerName="sequence-graph-0a095" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 21:27:31.771416 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:27:31.771081 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-0a095-6b6b78bdf9-crcwm" Jul 27 21:27:36.771275 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:27:36.771237 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-0a095-6b6b78bdf9-crcwm" podUID="aa6fb367-4009-42ea-b1b7-0d36f1f2aa0a" containerName="sequence-graph-0a095" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 21:27:41.771482 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:27:41.771440 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-0a095-6b6b78bdf9-crcwm" podUID="aa6fb367-4009-42ea-b1b7-0d36f1f2aa0a" containerName="sequence-graph-0a095" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 21:27:46.770546 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:27:46.770502 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-0a095-6b6b78bdf9-crcwm" podUID="aa6fb367-4009-42ea-b1b7-0d36f1f2aa0a" containerName="sequence-graph-0a095" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 21:27:49.942517 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:27:49.942477 2577 generic.go:356] "Generic (PLEG): container finished" podID="aa6fb367-4009-42ea-b1b7-0d36f1f2aa0a" containerID="811bb70ea3034bfeeceee36653877343ec05381fdf812719a0387aa61ba8d168" exitCode=0 Jul 27 21:27:49.942965 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:27:49.942551 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-0a095-6b6b78bdf9-crcwm" event={"ID":"aa6fb367-4009-42ea-b1b7-0d36f1f2aa0a","Type":"ContainerDied","Data":"811bb70ea3034bfeeceee36653877343ec05381fdf812719a0387aa61ba8d168"} Jul 27 21:27:50.056323 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:27:50.056289 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-0a095-6b6b78bdf9-crcwm" Jul 27 21:27:50.095385 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:27:50.095342 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa6fb367-4009-42ea-b1b7-0d36f1f2aa0a-proxy-tls\") pod \"aa6fb367-4009-42ea-b1b7-0d36f1f2aa0a\" (UID: \"aa6fb367-4009-42ea-b1b7-0d36f1f2aa0a\") " Jul 27 21:27:50.095548 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:27:50.095432 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa6fb367-4009-42ea-b1b7-0d36f1f2aa0a-openshift-service-ca-bundle\") pod \"aa6fb367-4009-42ea-b1b7-0d36f1f2aa0a\" (UID: \"aa6fb367-4009-42ea-b1b7-0d36f1f2aa0a\") " Jul 27 21:27:50.095760 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:27:50.095734 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa6fb367-4009-42ea-b1b7-0d36f1f2aa0a-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "aa6fb367-4009-42ea-b1b7-0d36f1f2aa0a" (UID: "aa6fb367-4009-42ea-b1b7-0d36f1f2aa0a"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 21:27:50.097414 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:27:50.097386 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa6fb367-4009-42ea-b1b7-0d36f1f2aa0a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "aa6fb367-4009-42ea-b1b7-0d36f1f2aa0a" (UID: "aa6fb367-4009-42ea-b1b7-0d36f1f2aa0a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 21:27:50.196139 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:27:50.196034 2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa6fb367-4009-42ea-b1b7-0d36f1f2aa0a-proxy-tls\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\"" Jul 27 21:27:50.196139 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:27:50.196067 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa6fb367-4009-42ea-b1b7-0d36f1f2aa0a-openshift-service-ca-bundle\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\"" Jul 27 21:27:50.948771 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:27:50.948737 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-0a095-6b6b78bdf9-crcwm" Jul 27 21:27:50.949250 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:27:50.948739 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-0a095-6b6b78bdf9-crcwm" event={"ID":"aa6fb367-4009-42ea-b1b7-0d36f1f2aa0a","Type":"ContainerDied","Data":"fbed982da7e3a10f8133fa8aa18a20b6be1313ad104051b3d003368b65ed8631"} Jul 27 21:27:50.949250 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:27:50.948894 2577 scope.go:117] "RemoveContainer" containerID="811bb70ea3034bfeeceee36653877343ec05381fdf812719a0387aa61ba8d168" Jul 27 21:27:50.965942 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:27:50.965909 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-0a095-6b6b78bdf9-crcwm"] Jul 27 21:27:50.969429 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:27:50.969404 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-0a095-6b6b78bdf9-crcwm"] Jul 27 21:27:52.365975 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:27:52.365941 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa6fb367-4009-42ea-b1b7-0d36f1f2aa0a" path="/var/lib/kubelet/pods/aa6fb367-4009-42ea-b1b7-0d36f1f2aa0a/volumes" Jul 27 21:28:19.630888 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:28:19.630842 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-0f5d7-785897b877-f6lpp"] Jul 27 21:28:19.631504 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:28:19.631342 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa6fb367-4009-42ea-b1b7-0d36f1f2aa0a" containerName="sequence-graph-0a095" Jul 27 21:28:19.631504 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:28:19.631363 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa6fb367-4009-42ea-b1b7-0d36f1f2aa0a" containerName="sequence-graph-0a095" Jul 27 21:28:19.631504 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:28:19.631402 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84913fee-eaea-4b13-b8df-d0c70596ad75" containerName="ensemble-graph-cb683" Jul 27 21:28:19.631504 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:28:19.631412 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="84913fee-eaea-4b13-b8df-d0c70596ad75" containerName="ensemble-graph-cb683" Jul 27 21:28:19.631504 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:28:19.631502 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="84913fee-eaea-4b13-b8df-d0c70596ad75" containerName="ensemble-graph-cb683" Jul 27 21:28:19.631802 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:28:19.631513 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa6fb367-4009-42ea-b1b7-0d36f1f2aa0a" containerName="sequence-graph-0a095" Jul 27 21:28:19.634651 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:28:19.634628 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-0f5d7-785897b877-f6lpp" Jul 27 21:28:19.638405 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:28:19.638374 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 27 21:28:19.638550 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:28:19.638430 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-0f5d7-serving-cert\"" Jul 27 21:28:19.639136 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:28:19.639120 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-0f5d7-kube-rbac-proxy-sar-config\"" Jul 27 21:28:19.639136 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:28:19.639128 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-wg596\"" Jul 27 21:28:19.648979 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:28:19.648948 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-0f5d7-785897b877-f6lpp"] Jul 27 21:28:19.751425 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:28:19.751384 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff147a09-6b60-4c76-b320-782ad945d971-proxy-tls\") pod \"sequence-graph-0f5d7-785897b877-f6lpp\" (UID: \"ff147a09-6b60-4c76-b320-782ad945d971\") " pod="kserve-ci-e2e-test/sequence-graph-0f5d7-785897b877-f6lpp" Jul 27 21:28:19.751616 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:28:19.751495 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff147a09-6b60-4c76-b320-782ad945d971-openshift-service-ca-bundle\") pod \"sequence-graph-0f5d7-785897b877-f6lpp\" (UID: \"ff147a09-6b60-4c76-b320-782ad945d971\") " pod="kserve-ci-e2e-test/sequence-graph-0f5d7-785897b877-f6lpp" Jul 27 21:28:19.852408 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:28:19.852360 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff147a09-6b60-4c76-b320-782ad945d971-openshift-service-ca-bundle\") pod \"sequence-graph-0f5d7-785897b877-f6lpp\" (UID: \"ff147a09-6b60-4c76-b320-782ad945d971\") " pod="kserve-ci-e2e-test/sequence-graph-0f5d7-785897b877-f6lpp" Jul 27 21:28:19.852636 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:28:19.852429 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff147a09-6b60-4c76-b320-782ad945d971-proxy-tls\") pod \"sequence-graph-0f5d7-785897b877-f6lpp\" (UID: \"ff147a09-6b60-4c76-b320-782ad945d971\") " pod="kserve-ci-e2e-test/sequence-graph-0f5d7-785897b877-f6lpp" Jul 27 21:28:19.852636 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:28:19.852533 2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-0f5d7-serving-cert: secret "sequence-graph-0f5d7-serving-cert" not found Jul 27 21:28:19.852636 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:28:19.852625 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff147a09-6b60-4c76-b320-782ad945d971-proxy-tls podName:ff147a09-6b60-4c76-b320-782ad945d971 nodeName:}" failed. No retries permitted until 2026-07-27 21:28:20.352609009 +0000 UTC m=+1492.484225211 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ff147a09-6b60-4c76-b320-782ad945d971-proxy-tls") pod "sequence-graph-0f5d7-785897b877-f6lpp" (UID: "ff147a09-6b60-4c76-b320-782ad945d971") : secret "sequence-graph-0f5d7-serving-cert" not found Jul 27 21:28:19.853040 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:28:19.853019 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff147a09-6b60-4c76-b320-782ad945d971-openshift-service-ca-bundle\") pod \"sequence-graph-0f5d7-785897b877-f6lpp\" (UID: \"ff147a09-6b60-4c76-b320-782ad945d971\") " pod="kserve-ci-e2e-test/sequence-graph-0f5d7-785897b877-f6lpp" Jul 27 21:28:20.357721 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:28:20.357666 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff147a09-6b60-4c76-b320-782ad945d971-proxy-tls\") pod \"sequence-graph-0f5d7-785897b877-f6lpp\" (UID: \"ff147a09-6b60-4c76-b320-782ad945d971\") " pod="kserve-ci-e2e-test/sequence-graph-0f5d7-785897b877-f6lpp" Jul 27 21:28:20.360155 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:28:20.360127 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff147a09-6b60-4c76-b320-782ad945d971-proxy-tls\") pod \"sequence-graph-0f5d7-785897b877-f6lpp\" (UID: \"ff147a09-6b60-4c76-b320-782ad945d971\") " pod="kserve-ci-e2e-test/sequence-graph-0f5d7-785897b877-f6lpp" Jul 27 21:28:20.545105 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:28:20.545062 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-0f5d7-785897b877-f6lpp" Jul 27 21:28:20.683268 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:28:20.683241 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-0f5d7-785897b877-f6lpp"] Jul 27 21:28:20.685905 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:28:20.685876 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff147a09_6b60_4c76_b320_782ad945d971.slice/crio-a50a495151df92487447adf5e8410c9aa2b1731fb76baa9340a04e0331f55381 WatchSource:0}: Error finding container a50a495151df92487447adf5e8410c9aa2b1731fb76baa9340a04e0331f55381: Status 404 returned error can't find the container with id a50a495151df92487447adf5e8410c9aa2b1731fb76baa9340a04e0331f55381 Jul 27 21:28:21.085965 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:28:21.085928 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-0f5d7-785897b877-f6lpp" event={"ID":"ff147a09-6b60-4c76-b320-782ad945d971","Type":"ContainerStarted","Data":"2ab5563fc0261e7e833f52195c3c07729370942a4d5bc774681b7b4f776368b5"} Jul 27 21:28:21.086142 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:28:21.085984 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-0f5d7-785897b877-f6lpp" event={"ID":"ff147a09-6b60-4c76-b320-782ad945d971","Type":"ContainerStarted","Data":"a50a495151df92487447adf5e8410c9aa2b1731fb76baa9340a04e0331f55381"} Jul 27 21:28:21.086142 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:28:21.086070 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-0f5d7-785897b877-f6lpp" Jul 27 21:28:21.102720 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:28:21.102668 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-0f5d7-785897b877-f6lpp" podStartSLOduration=2.102652829 podStartE2EDuration="2.102652829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:28:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:28:21.100940347 +0000 UTC m=+1493.232556570" watchObservedRunningTime="2026-07-27 21:28:21.102652829 +0000 UTC m=+1493.234269052" Jul 27 21:28:27.096698 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:28:27.096664 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-0f5d7-785897b877-f6lpp" Jul 27 21:28:28.353030 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:28:28.352999 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-j5rmn_4812c628-5f41-4e36-aaf6-8d62e4b6a443/console-operator/2.log" Jul 27 21:28:28.354274 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:28:28.354248 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-j5rmn_4812c628-5f41-4e36-aaf6-8d62e4b6a443/console-operator/2.log" Jul 27 21:28:28.357661 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:28:28.357644 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6mhn_1907dc14-1132-4eb9-a5ee-675b43df6e6d/ovn-acl-logging/0.log" Jul 27 21:28:28.358881 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:28:28.358858 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6mhn_1907dc14-1132-4eb9-a5ee-675b43df6e6d/ovn-acl-logging/0.log" Jul 27 21:33:28.387201 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:33:28.387165 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-j5rmn_4812c628-5f41-4e36-aaf6-8d62e4b6a443/console-operator/2.log" Jul 27 21:33:28.387911 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:33:28.387891 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-j5rmn_4812c628-5f41-4e36-aaf6-8d62e4b6a443/console-operator/2.log" Jul 27 21:33:28.391939 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:33:28.391917 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6mhn_1907dc14-1132-4eb9-a5ee-675b43df6e6d/ovn-acl-logging/0.log" Jul 27 21:33:28.392918 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:33:28.392889 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6mhn_1907dc14-1132-4eb9-a5ee-675b43df6e6d/ovn-acl-logging/0.log" Jul 27 21:36:34.283904 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:36:34.283811 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-0f5d7-785897b877-f6lpp"] Jul 27 21:36:34.284449 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:36:34.284019 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-0f5d7-785897b877-f6lpp" podUID="ff147a09-6b60-4c76-b320-782ad945d971" containerName="sequence-graph-0f5d7" containerID="cri-o://2ab5563fc0261e7e833f52195c3c07729370942a4d5bc774681b7b4f776368b5" gracePeriod=30 Jul 27 21:36:37.094240 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:36:37.094200 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-0f5d7-785897b877-f6lpp" podUID="ff147a09-6b60-4c76-b320-782ad945d971" containerName="sequence-graph-0f5d7" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 21:36:42.094000 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:36:42.093958 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-0f5d7-785897b877-f6lpp" podUID="ff147a09-6b60-4c76-b320-782ad945d971" containerName="sequence-graph-0f5d7" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 21:36:47.094234 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:36:47.094194 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-0f5d7-785897b877-f6lpp" podUID="ff147a09-6b60-4c76-b320-782ad945d971" containerName="sequence-graph-0f5d7" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 21:36:47.094661 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:36:47.094338 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-0f5d7-785897b877-f6lpp" Jul 27 21:36:52.094509 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:36:52.094463 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-0f5d7-785897b877-f6lpp" podUID="ff147a09-6b60-4c76-b320-782ad945d971" containerName="sequence-graph-0f5d7" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 21:36:57.094723 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:36:57.094679 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-0f5d7-785897b877-f6lpp" podUID="ff147a09-6b60-4c76-b320-782ad945d971" containerName="sequence-graph-0f5d7" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 21:37:02.093886 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:37:02.093846 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-0f5d7-785897b877-f6lpp" podUID="ff147a09-6b60-4c76-b320-782ad945d971" containerName="sequence-graph-0f5d7" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 21:37:04.449766 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:37:04.449737 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-0f5d7-785897b877-f6lpp" Jul 27 21:37:04.451846 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:37:04.451819 2577 generic.go:356] "Generic (PLEG): container finished" podID="ff147a09-6b60-4c76-b320-782ad945d971" containerID="2ab5563fc0261e7e833f52195c3c07729370942a4d5bc774681b7b4f776368b5" exitCode=0 Jul 27 21:37:04.451973 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:37:04.451855 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-0f5d7-785897b877-f6lpp" Jul 27 21:37:04.451973 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:37:04.451912 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-0f5d7-785897b877-f6lpp" event={"ID":"ff147a09-6b60-4c76-b320-782ad945d971","Type":"ContainerDied","Data":"2ab5563fc0261e7e833f52195c3c07729370942a4d5bc774681b7b4f776368b5"} Jul 27 21:37:04.452085 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:37:04.451980 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-0f5d7-785897b877-f6lpp" event={"ID":"ff147a09-6b60-4c76-b320-782ad945d971","Type":"ContainerDied","Data":"a50a495151df92487447adf5e8410c9aa2b1731fb76baa9340a04e0331f55381"} Jul 27 21:37:04.452085 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:37:04.452014 2577 scope.go:117] "RemoveContainer" containerID="2ab5563fc0261e7e833f52195c3c07729370942a4d5bc774681b7b4f776368b5" Jul 27 21:37:04.460533 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:37:04.460512 2577 scope.go:117] "RemoveContainer" containerID="2ab5563fc0261e7e833f52195c3c07729370942a4d5bc774681b7b4f776368b5" Jul 27 21:37:04.460860 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:37:04.460832 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ab5563fc0261e7e833f52195c3c07729370942a4d5bc774681b7b4f776368b5\": container with ID starting with 2ab5563fc0261e7e833f52195c3c07729370942a4d5bc774681b7b4f776368b5 not found: ID does not exist" containerID="2ab5563fc0261e7e833f52195c3c07729370942a4d5bc774681b7b4f776368b5" Jul 27 21:37:04.460944 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:37:04.460862 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ab5563fc0261e7e833f52195c3c07729370942a4d5bc774681b7b4f776368b5"} err="failed to get container status \"2ab5563fc0261e7e833f52195c3c07729370942a4d5bc774681b7b4f776368b5\": rpc error: code = NotFound desc = could not find container \"2ab5563fc0261e7e833f52195c3c07729370942a4d5bc774681b7b4f776368b5\": container with ID starting with 2ab5563fc0261e7e833f52195c3c07729370942a4d5bc774681b7b4f776368b5 not found: ID does not exist" Jul 27 21:37:04.556448 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:37:04.556353 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff147a09-6b60-4c76-b320-782ad945d971-proxy-tls\") pod \"ff147a09-6b60-4c76-b320-782ad945d971\" (UID: \"ff147a09-6b60-4c76-b320-782ad945d971\") " Jul 27 21:37:04.556638 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:37:04.556457 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff147a09-6b60-4c76-b320-782ad945d971-openshift-service-ca-bundle\") pod \"ff147a09-6b60-4c76-b320-782ad945d971\" (UID: \"ff147a09-6b60-4c76-b320-782ad945d971\") " Jul 27 21:37:04.556870 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:37:04.556845 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff147a09-6b60-4c76-b320-782ad945d971-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "ff147a09-6b60-4c76-b320-782ad945d971" (UID: "ff147a09-6b60-4c76-b320-782ad945d971"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 21:37:04.558499 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:37:04.558473 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff147a09-6b60-4c76-b320-782ad945d971-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ff147a09-6b60-4c76-b320-782ad945d971" (UID: "ff147a09-6b60-4c76-b320-782ad945d971"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 21:37:04.657014 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:37:04.656978 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff147a09-6b60-4c76-b320-782ad945d971-openshift-service-ca-bundle\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\"" Jul 27 21:37:04.657014 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:37:04.657007 2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff147a09-6b60-4c76-b320-782ad945d971-proxy-tls\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\"" Jul 27 21:37:04.777638 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:37:04.777599 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-61561-79f8d69754-l7866"] Jul 27 21:37:04.777957 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:37:04.777942 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff147a09-6b60-4c76-b320-782ad945d971" containerName="sequence-graph-0f5d7" Jul 27 21:37:04.778004 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:37:04.777959 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff147a09-6b60-4c76-b320-782ad945d971" containerName="sequence-graph-0f5d7" Jul 27 21:37:04.778038 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:37:04.778014 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff147a09-6b60-4c76-b320-782ad945d971" containerName="sequence-graph-0f5d7" Jul 27 21:37:04.782144 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:37:04.782121 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-0f5d7-785897b877-f6lpp"] Jul 27 21:37:04.782144 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:37:04.782149 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-0f5d7-785897b877-f6lpp"] Jul 27 21:37:04.782301 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:37:04.782234 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-61561-79f8d69754-l7866" Jul 27 21:37:04.784366 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:37:04.784342 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-61561-serving-cert\"" Jul 27 21:37:04.784492 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:37:04.784407 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 27 21:37:04.784582 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:37:04.784548 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-61561-kube-rbac-proxy-sar-config\"" Jul 27 21:37:04.784676 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:37:04.784644 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-wg596\"" Jul 27 21:37:04.792594 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:37:04.792554 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-61561-79f8d69754-l7866"] Jul 27 21:37:04.858633 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:37:04.858542 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2793424c-e52e-4298-bace-85e9689ace53-proxy-tls\") pod \"splitter-graph-61561-79f8d69754-l7866\" (UID: \"2793424c-e52e-4298-bace-85e9689ace53\") " pod="kserve-ci-e2e-test/splitter-graph-61561-79f8d69754-l7866" Jul 27 21:37:04.858633 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:37:04.858610 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2793424c-e52e-4298-bace-85e9689ace53-openshift-service-ca-bundle\") pod \"splitter-graph-61561-79f8d69754-l7866\" (UID: \"2793424c-e52e-4298-bace-85e9689ace53\") " pod="kserve-ci-e2e-test/splitter-graph-61561-79f8d69754-l7866" Jul 27 21:37:04.959195 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:37:04.959159 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2793424c-e52e-4298-bace-85e9689ace53-proxy-tls\") pod \"splitter-graph-61561-79f8d69754-l7866\" (UID: \"2793424c-e52e-4298-bace-85e9689ace53\") " pod="kserve-ci-e2e-test/splitter-graph-61561-79f8d69754-l7866" Jul 27 21:37:04.959195 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:37:04.959196 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2793424c-e52e-4298-bace-85e9689ace53-openshift-service-ca-bundle\") pod \"splitter-graph-61561-79f8d69754-l7866\" (UID: \"2793424c-e52e-4298-bace-85e9689ace53\") " pod="kserve-ci-e2e-test/splitter-graph-61561-79f8d69754-l7866" Jul 27 21:37:04.959865 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:37:04.959843 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2793424c-e52e-4298-bace-85e9689ace53-openshift-service-ca-bundle\") pod \"splitter-graph-61561-79f8d69754-l7866\" (UID: \"2793424c-e52e-4298-bace-85e9689ace53\") " pod="kserve-ci-e2e-test/splitter-graph-61561-79f8d69754-l7866" Jul 27 21:37:04.961620 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:37:04.961599 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2793424c-e52e-4298-bace-85e9689ace53-proxy-tls\") pod \"splitter-graph-61561-79f8d69754-l7866\" (UID: \"2793424c-e52e-4298-bace-85e9689ace53\") " pod="kserve-ci-e2e-test/splitter-graph-61561-79f8d69754-l7866" Jul 27 21:37:05.094170 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:37:05.094133 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-61561-79f8d69754-l7866" Jul 27 21:37:05.229630 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:37:05.229600 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-61561-79f8d69754-l7866"] Jul 27 21:37:05.232145 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:37:05.232110 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2793424c_e52e_4298_bace_85e9689ace53.slice/crio-6e8ac13d467b875ff2aee5c070ad5854099a81e6acfc6fe712aa9fae9ee9cec6 WatchSource:0}: Error finding container 6e8ac13d467b875ff2aee5c070ad5854099a81e6acfc6fe712aa9fae9ee9cec6: Status 404 returned error can't find the container with id 6e8ac13d467b875ff2aee5c070ad5854099a81e6acfc6fe712aa9fae9ee9cec6 Jul 27 21:37:05.458973 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:37:05.458879 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-61561-79f8d69754-l7866" event={"ID":"2793424c-e52e-4298-bace-85e9689ace53","Type":"ContainerStarted","Data":"954437ddc50c345c3d8580ae2bbbd936230c45388c8e53478180a52603ca5c76"} Jul 27 21:37:05.458973 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:37:05.458930 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-61561-79f8d69754-l7866" event={"ID":"2793424c-e52e-4298-bace-85e9689ace53","Type":"ContainerStarted","Data":"6e8ac13d467b875ff2aee5c070ad5854099a81e6acfc6fe712aa9fae9ee9cec6"} Jul 27 21:37:05.459419 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:37:05.459074 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-61561-79f8d69754-l7866" Jul 27 21:37:05.475856 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:37:05.475801 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-61561-79f8d69754-l7866" podStartSLOduration=1.475784811 podStartE2EDuration="1.475784811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:37:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:37:05.475204362 +0000 UTC m=+2017.606820584" watchObservedRunningTime="2026-07-27 21:37:05.475784811 +0000 UTC m=+2017.607401034" Jul 27 21:37:06.366797 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:37:06.366768 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff147a09-6b60-4c76-b320-782ad945d971" path="/var/lib/kubelet/pods/ff147a09-6b60-4c76-b320-782ad945d971/volumes" Jul 27 21:37:11.468926 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:37:11.468893 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-61561-79f8d69754-l7866" Jul 27 21:37:14.852005 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:37:14.851916 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-61561-79f8d69754-l7866"] Jul 27 21:37:14.852407 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:37:14.852117 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-61561-79f8d69754-l7866" podUID="2793424c-e52e-4298-bace-85e9689ace53" containerName="splitter-graph-61561" containerID="cri-o://954437ddc50c345c3d8580ae2bbbd936230c45388c8e53478180a52603ca5c76" gracePeriod=30 Jul 27 21:37:16.467180 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:37:16.467139 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-61561-79f8d69754-l7866" podUID="2793424c-e52e-4298-bace-85e9689ace53" containerName="splitter-graph-61561" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 21:37:21.467304 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:37:21.467261 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-61561-79f8d69754-l7866" podUID="2793424c-e52e-4298-bace-85e9689ace53" containerName="splitter-graph-61561" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 21:37:26.467252 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:37:26.467208 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-61561-79f8d69754-l7866" podUID="2793424c-e52e-4298-bace-85e9689ace53" containerName="splitter-graph-61561" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 21:37:26.467673 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:37:26.467338 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-61561-79f8d69754-l7866" Jul 27 21:37:31.467879 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:37:31.467834 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-61561-79f8d69754-l7866" podUID="2793424c-e52e-4298-bace-85e9689ace53" containerName="splitter-graph-61561" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 21:37:36.467097 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:37:36.467047 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-61561-79f8d69754-l7866" podUID="2793424c-e52e-4298-bace-85e9689ace53" containerName="splitter-graph-61561" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 21:37:41.467483 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:37:41.467436 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-61561-79f8d69754-l7866" podUID="2793424c-e52e-4298-bace-85e9689ace53" containerName="splitter-graph-61561" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 21:37:45.010403 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:37:45.010375 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-61561-79f8d69754-l7866" Jul 27 21:37:45.122284 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:37:45.122192 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2793424c-e52e-4298-bace-85e9689ace53-proxy-tls\") pod \"2793424c-e52e-4298-bace-85e9689ace53\" (UID: \"2793424c-e52e-4298-bace-85e9689ace53\") " Jul 27 21:37:45.122284 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:37:45.122262 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2793424c-e52e-4298-bace-85e9689ace53-openshift-service-ca-bundle\") pod \"2793424c-e52e-4298-bace-85e9689ace53\" (UID: \"2793424c-e52e-4298-bace-85e9689ace53\") " Jul 27 21:37:45.122726 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:37:45.122690 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2793424c-e52e-4298-bace-85e9689ace53-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "2793424c-e52e-4298-bace-85e9689ace53" (UID: "2793424c-e52e-4298-bace-85e9689ace53"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 21:37:45.124336 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:37:45.124313 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2793424c-e52e-4298-bace-85e9689ace53-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2793424c-e52e-4298-bace-85e9689ace53" (UID: "2793424c-e52e-4298-bace-85e9689ace53"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 21:37:45.222794 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:37:45.222759 2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2793424c-e52e-4298-bace-85e9689ace53-proxy-tls\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\"" Jul 27 21:37:45.222794 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:37:45.222788 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2793424c-e52e-4298-bace-85e9689ace53-openshift-service-ca-bundle\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\"" Jul 27 21:37:45.642634 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:37:45.642595 2577 generic.go:356] "Generic (PLEG): container finished" podID="2793424c-e52e-4298-bace-85e9689ace53" containerID="954437ddc50c345c3d8580ae2bbbd936230c45388c8e53478180a52603ca5c76" exitCode=0 Jul 27 21:37:45.642634 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:37:45.642636 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-61561-79f8d69754-l7866" Jul 27 21:37:45.642852 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:37:45.642668 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-61561-79f8d69754-l7866" event={"ID":"2793424c-e52e-4298-bace-85e9689ace53","Type":"ContainerDied","Data":"954437ddc50c345c3d8580ae2bbbd936230c45388c8e53478180a52603ca5c76"} Jul 27 21:37:45.642852 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:37:45.642725 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-61561-79f8d69754-l7866" event={"ID":"2793424c-e52e-4298-bace-85e9689ace53","Type":"ContainerDied","Data":"6e8ac13d467b875ff2aee5c070ad5854099a81e6acfc6fe712aa9fae9ee9cec6"} Jul 27 21:37:45.642852 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:37:45.642755 2577 scope.go:117] "RemoveContainer" containerID="954437ddc50c345c3d8580ae2bbbd936230c45388c8e53478180a52603ca5c76" Jul 27 21:37:45.651651 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:37:45.651629 2577 scope.go:117] "RemoveContainer" containerID="954437ddc50c345c3d8580ae2bbbd936230c45388c8e53478180a52603ca5c76" Jul 27 21:37:45.651938 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:37:45.651918 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"954437ddc50c345c3d8580ae2bbbd936230c45388c8e53478180a52603ca5c76\": container with ID starting with 954437ddc50c345c3d8580ae2bbbd936230c45388c8e53478180a52603ca5c76 not found: ID does not exist" containerID="954437ddc50c345c3d8580ae2bbbd936230c45388c8e53478180a52603ca5c76" Jul 27 21:37:45.652006 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:37:45.651946 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"954437ddc50c345c3d8580ae2bbbd936230c45388c8e53478180a52603ca5c76"} err="failed to get container status \"954437ddc50c345c3d8580ae2bbbd936230c45388c8e53478180a52603ca5c76\": rpc error: code = NotFound desc = could not find container \"954437ddc50c345c3d8580ae2bbbd936230c45388c8e53478180a52603ca5c76\": container with ID starting with 954437ddc50c345c3d8580ae2bbbd936230c45388c8e53478180a52603ca5c76 not found: ID does not exist" Jul 27 21:37:45.663609 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:37:45.663563 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-61561-79f8d69754-l7866"] Jul 27 21:37:45.668087 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:37:45.668061 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-61561-79f8d69754-l7866"] Jul 27 21:37:46.366540 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:37:46.366508 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2793424c-e52e-4298-bace-85e9689ace53" path="/var/lib/kubelet/pods/2793424c-e52e-4298-bace-85e9689ace53/volumes" Jul 27 21:38:15.088429 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:38:15.088394 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-b2e96-c4cf9c888-47dlv"] Jul 27 21:38:15.088866 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:38:15.088735 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2793424c-e52e-4298-bace-85e9689ace53" containerName="splitter-graph-61561" Jul 27 21:38:15.088866 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:38:15.088761 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2793424c-e52e-4298-bace-85e9689ace53" containerName="splitter-graph-61561" Jul 27 21:38:15.088866 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:38:15.088825 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2793424c-e52e-4298-bace-85e9689ace53" containerName="splitter-graph-61561" Jul 27 21:38:15.091832 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:38:15.091810 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-b2e96-c4cf9c888-47dlv" Jul 27 21:38:15.094159 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:38:15.094129 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-b2e96-kube-rbac-proxy-sar-config\"" Jul 27 21:38:15.094423 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:38:15.094404 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-b2e96-serving-cert\"" Jul 27 21:38:15.095087 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:38:15.095068 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 27 21:38:15.095214 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:38:15.095157 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-wg596\"" Jul 27 21:38:15.101739 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:38:15.101712 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-b2e96-c4cf9c888-47dlv"] Jul 27 21:38:15.174336 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:38:15.174295 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3801fd76-3df5-4ae1-a917-62e1f490c3fe-proxy-tls\") pod \"splitter-graph-b2e96-c4cf9c888-47dlv\" (UID: \"3801fd76-3df5-4ae1-a917-62e1f490c3fe\") " pod="kserve-ci-e2e-test/splitter-graph-b2e96-c4cf9c888-47dlv" Jul 27 21:38:15.174531 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:38:15.174434 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3801fd76-3df5-4ae1-a917-62e1f490c3fe-openshift-service-ca-bundle\") pod \"splitter-graph-b2e96-c4cf9c888-47dlv\" (UID: \"3801fd76-3df5-4ae1-a917-62e1f490c3fe\") " pod="kserve-ci-e2e-test/splitter-graph-b2e96-c4cf9c888-47dlv" Jul 27 21:38:15.275916 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:38:15.275869 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3801fd76-3df5-4ae1-a917-62e1f490c3fe-proxy-tls\") pod \"splitter-graph-b2e96-c4cf9c888-47dlv\" (UID: \"3801fd76-3df5-4ae1-a917-62e1f490c3fe\") " pod="kserve-ci-e2e-test/splitter-graph-b2e96-c4cf9c888-47dlv" Jul 27 21:38:15.276110 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:38:15.275948 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3801fd76-3df5-4ae1-a917-62e1f490c3fe-openshift-service-ca-bundle\") pod \"splitter-graph-b2e96-c4cf9c888-47dlv\" (UID: \"3801fd76-3df5-4ae1-a917-62e1f490c3fe\") " pod="kserve-ci-e2e-test/splitter-graph-b2e96-c4cf9c888-47dlv" Jul 27 21:38:15.276794 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:38:15.276773 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3801fd76-3df5-4ae1-a917-62e1f490c3fe-openshift-service-ca-bundle\") pod \"splitter-graph-b2e96-c4cf9c888-47dlv\" (UID: \"3801fd76-3df5-4ae1-a917-62e1f490c3fe\") " pod="kserve-ci-e2e-test/splitter-graph-b2e96-c4cf9c888-47dlv" Jul 27 21:38:15.278491 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:38:15.278453 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3801fd76-3df5-4ae1-a917-62e1f490c3fe-proxy-tls\") pod \"splitter-graph-b2e96-c4cf9c888-47dlv\" (UID: \"3801fd76-3df5-4ae1-a917-62e1f490c3fe\") " pod="kserve-ci-e2e-test/splitter-graph-b2e96-c4cf9c888-47dlv" Jul 27 21:38:15.404032 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:38:15.403932 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-b2e96-c4cf9c888-47dlv" Jul 27 21:38:15.537660 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:38:15.537532 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-b2e96-c4cf9c888-47dlv"] Jul 27 21:38:15.540460 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:38:15.540431 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3801fd76_3df5_4ae1_a917_62e1f490c3fe.slice/crio-ccce10f87820262263f32d3e6b476358f48bd01bbe1956c644c0d3aa00709d03 WatchSource:0}: Error finding container ccce10f87820262263f32d3e6b476358f48bd01bbe1956c644c0d3aa00709d03: Status 404 returned error can't find the container with id ccce10f87820262263f32d3e6b476358f48bd01bbe1956c644c0d3aa00709d03 Jul 27 21:38:15.777234 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:38:15.777196 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-b2e96-c4cf9c888-47dlv" event={"ID":"3801fd76-3df5-4ae1-a917-62e1f490c3fe","Type":"ContainerStarted","Data":"55290bd44e209bcd002aad3f5022940c74cebca13612b57a6b9bce1cfe88b652"} Jul 27 21:38:15.777435 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:38:15.777263 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-b2e96-c4cf9c888-47dlv" event={"ID":"3801fd76-3df5-4ae1-a917-62e1f490c3fe","Type":"ContainerStarted","Data":"ccce10f87820262263f32d3e6b476358f48bd01bbe1956c644c0d3aa00709d03"} Jul 27 21:38:15.777435 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:38:15.777392 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-b2e96-c4cf9c888-47dlv" Jul 27 21:38:15.794181 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:38:15.794128 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-b2e96-c4cf9c888-47dlv" podStartSLOduration=0.794113132 podStartE2EDuration="794.113132ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:38:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:38:15.793483884 +0000 UTC m=+2087.925100107" watchObservedRunningTime="2026-07-27 21:38:15.794113132 +0000 UTC m=+2087.925729355" Jul 27 21:38:21.787712 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:38:21.787678 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-b2e96-c4cf9c888-47dlv" Jul 27 21:38:28.414474 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:38:28.414442 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-j5rmn_4812c628-5f41-4e36-aaf6-8d62e4b6a443/console-operator/2.log" Jul 27 21:38:28.416786 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:38:28.416759 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-j5rmn_4812c628-5f41-4e36-aaf6-8d62e4b6a443/console-operator/2.log" Jul 27 21:38:28.419056 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:38:28.419035 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6mhn_1907dc14-1132-4eb9-a5ee-675b43df6e6d/ovn-acl-logging/0.log" Jul 27 21:38:28.421319 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:38:28.421300 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6mhn_1907dc14-1132-4eb9-a5ee-675b43df6e6d/ovn-acl-logging/0.log" Jul 27 21:43:28.441406 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:43:28.441365 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-j5rmn_4812c628-5f41-4e36-aaf6-8d62e4b6a443/console-operator/2.log" Jul 27 21:43:28.445131 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:43:28.445106 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-j5rmn_4812c628-5f41-4e36-aaf6-8d62e4b6a443/console-operator/2.log" Jul 27 21:43:28.446181 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:43:28.446160 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6mhn_1907dc14-1132-4eb9-a5ee-675b43df6e6d/ovn-acl-logging/0.log" Jul 27 21:43:28.449905 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:43:28.449885 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6mhn_1907dc14-1132-4eb9-a5ee-675b43df6e6d/ovn-acl-logging/0.log" Jul 27 21:46:29.676853 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:46:29.676818 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-b2e96-c4cf9c888-47dlv"] Jul 27 21:46:29.677427 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:46:29.677120 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-b2e96-c4cf9c888-47dlv" podUID="3801fd76-3df5-4ae1-a917-62e1f490c3fe" containerName="splitter-graph-b2e96" containerID="cri-o://55290bd44e209bcd002aad3f5022940c74cebca13612b57a6b9bce1cfe88b652" gracePeriod=30 Jul 27 21:46:31.785791 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:46:31.785747 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-b2e96-c4cf9c888-47dlv" podUID="3801fd76-3df5-4ae1-a917-62e1f490c3fe" containerName="splitter-graph-b2e96" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 21:46:36.785953 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:46:36.785908 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-b2e96-c4cf9c888-47dlv" podUID="3801fd76-3df5-4ae1-a917-62e1f490c3fe" containerName="splitter-graph-b2e96" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 21:46:41.786471 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:46:41.786431 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-b2e96-c4cf9c888-47dlv" podUID="3801fd76-3df5-4ae1-a917-62e1f490c3fe" containerName="splitter-graph-b2e96" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 21:46:41.786909 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:46:41.786535 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-b2e96-c4cf9c888-47dlv" Jul 27 21:46:46.785776 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:46:46.785734 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-b2e96-c4cf9c888-47dlv" podUID="3801fd76-3df5-4ae1-a917-62e1f490c3fe" containerName="splitter-graph-b2e96" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 21:46:51.786324 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:46:51.786214 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-b2e96-c4cf9c888-47dlv" podUID="3801fd76-3df5-4ae1-a917-62e1f490c3fe" containerName="splitter-graph-b2e96" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 21:46:56.786437 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:46:56.786395 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-b2e96-c4cf9c888-47dlv" podUID="3801fd76-3df5-4ae1-a917-62e1f490c3fe" containerName="splitter-graph-b2e96" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 21:46:59.829433 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:46:59.829405 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-b2e96-c4cf9c888-47dlv" Jul 27 21:46:59.971059 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:46:59.971028 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3801fd76-3df5-4ae1-a917-62e1f490c3fe-proxy-tls\") pod \"3801fd76-3df5-4ae1-a917-62e1f490c3fe\" (UID: \"3801fd76-3df5-4ae1-a917-62e1f490c3fe\") " Jul 27 21:46:59.971213 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:46:59.971071 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3801fd76-3df5-4ae1-a917-62e1f490c3fe-openshift-service-ca-bundle\") pod \"3801fd76-3df5-4ae1-a917-62e1f490c3fe\" (UID: \"3801fd76-3df5-4ae1-a917-62e1f490c3fe\") " Jul 27 21:46:59.971440 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:46:59.971417 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3801fd76-3df5-4ae1-a917-62e1f490c3fe-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "3801fd76-3df5-4ae1-a917-62e1f490c3fe" (UID: "3801fd76-3df5-4ae1-a917-62e1f490c3fe"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 21:46:59.973260 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:46:59.973238 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3801fd76-3df5-4ae1-a917-62e1f490c3fe-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3801fd76-3df5-4ae1-a917-62e1f490c3fe" (UID: "3801fd76-3df5-4ae1-a917-62e1f490c3fe"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 21:47:00.072400 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:47:00.072357 2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3801fd76-3df5-4ae1-a917-62e1f490c3fe-proxy-tls\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\"" Jul 27 21:47:00.072400 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:47:00.072391 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3801fd76-3df5-4ae1-a917-62e1f490c3fe-openshift-service-ca-bundle\") on node \"ip-10-0-128-201.ec2.internal\" DevicePath \"\"" Jul 27 21:47:00.110911 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:47:00.110878 2577 generic.go:356] "Generic (PLEG): container finished" podID="3801fd76-3df5-4ae1-a917-62e1f490c3fe" containerID="55290bd44e209bcd002aad3f5022940c74cebca13612b57a6b9bce1cfe88b652" exitCode=0 Jul 27 21:47:00.110911 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:47:00.110916 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-b2e96-c4cf9c888-47dlv" event={"ID":"3801fd76-3df5-4ae1-a917-62e1f490c3fe","Type":"ContainerDied","Data":"55290bd44e209bcd002aad3f5022940c74cebca13612b57a6b9bce1cfe88b652"} Jul 27 21:47:00.111117 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:47:00.110930 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-b2e96-c4cf9c888-47dlv" Jul 27 21:47:00.111117 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:47:00.110958 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-b2e96-c4cf9c888-47dlv" event={"ID":"3801fd76-3df5-4ae1-a917-62e1f490c3fe","Type":"ContainerDied","Data":"ccce10f87820262263f32d3e6b476358f48bd01bbe1956c644c0d3aa00709d03"} Jul 27 21:47:00.111117 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:47:00.110979 2577 scope.go:117] "RemoveContainer" containerID="55290bd44e209bcd002aad3f5022940c74cebca13612b57a6b9bce1cfe88b652" Jul 27 21:47:00.119944 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:47:00.119922 2577 scope.go:117] "RemoveContainer" containerID="55290bd44e209bcd002aad3f5022940c74cebca13612b57a6b9bce1cfe88b652" Jul 27 21:47:00.120231 ip-10-0-128-201 kubenswrapper[2577]: E0727 21:47:00.120209 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55290bd44e209bcd002aad3f5022940c74cebca13612b57a6b9bce1cfe88b652\": container with ID starting with 55290bd44e209bcd002aad3f5022940c74cebca13612b57a6b9bce1cfe88b652 not found: ID does not exist" containerID="55290bd44e209bcd002aad3f5022940c74cebca13612b57a6b9bce1cfe88b652" Jul 27 21:47:00.120280 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:47:00.120254 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55290bd44e209bcd002aad3f5022940c74cebca13612b57a6b9bce1cfe88b652"} err="failed to get container status \"55290bd44e209bcd002aad3f5022940c74cebca13612b57a6b9bce1cfe88b652\": rpc error: code = NotFound desc = could not find container \"55290bd44e209bcd002aad3f5022940c74cebca13612b57a6b9bce1cfe88b652\": container with ID starting with 55290bd44e209bcd002aad3f5022940c74cebca13612b57a6b9bce1cfe88b652 not found: ID does not exist" Jul 27 21:47:00.131744 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:47:00.131709 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-b2e96-c4cf9c888-47dlv"] Jul 27 21:47:00.135428 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:47:00.135402 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-b2e96-c4cf9c888-47dlv"] Jul 27 21:47:00.367333 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:47:00.367255 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3801fd76-3df5-4ae1-a917-62e1f490c3fe" path="/var/lib/kubelet/pods/3801fd76-3df5-4ae1-a917-62e1f490c3fe/volumes" Jul 27 21:48:28.467483 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:48:28.467378 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-j5rmn_4812c628-5f41-4e36-aaf6-8d62e4b6a443/console-operator/2.log" Jul 27 21:48:28.472139 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:48:28.472121 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6mhn_1907dc14-1132-4eb9-a5ee-675b43df6e6d/ovn-acl-logging/0.log" Jul 27 21:48:28.472269 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:48:28.472171 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-j5rmn_4812c628-5f41-4e36-aaf6-8d62e4b6a443/console-operator/2.log" Jul 27 21:48:28.476835 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:48:28.476814 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6mhn_1907dc14-1132-4eb9-a5ee-675b43df6e6d/ovn-acl-logging/0.log" Jul 27 21:53:28.495084 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:53:28.495049 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-j5rmn_4812c628-5f41-4e36-aaf6-8d62e4b6a443/console-operator/2.log" Jul 27 21:53:28.500021 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:53:28.499994 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-j5rmn_4812c628-5f41-4e36-aaf6-8d62e4b6a443/console-operator/2.log" Jul 27 21:53:28.500179 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:53:28.500029 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6mhn_1907dc14-1132-4eb9-a5ee-675b43df6e6d/ovn-acl-logging/0.log" Jul 27 21:53:28.504403 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:53:28.504380 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6mhn_1907dc14-1132-4eb9-a5ee-675b43df6e6d/ovn-acl-logging/0.log" Jul 27 21:54:27.120690 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:27.120587 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-l2czx_c77555ba-462e-49c5-b8b0-e83987e8a2e3/global-pull-secret-syncer/0.log" Jul 27 21:54:27.292933 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:27.292896 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-tx4cs_de32d9f8-5f7d-4a68-8aa4-2830c11db8f5/konnectivity-agent/0.log" Jul 27 21:54:27.315237 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:27.315202 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-201.ec2.internal_4572a1a28313da7538ca6395c7dccdb4/haproxy/0.log" Jul 27 21:54:31.559198 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:31.559162 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-dm69z_91a29cde-048a-4e5e-a804-5854f86081ed/cluster-monitoring-operator/0.log" Jul 27 21:54:31.807770 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:31.807733 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-q9q9v_f0119fc2-7a0d-4f10-ae6a-e583297b08e4/node-exporter/0.log" Jul 27 21:54:31.828107 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:31.828023 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-q9q9v_f0119fc2-7a0d-4f10-ae6a-e583297b08e4/kube-rbac-proxy/0.log" Jul 27 21:54:31.849936 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:31.849907 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-q9q9v_f0119fc2-7a0d-4f10-ae6a-e583297b08e4/init-textfile/0.log" Jul 27 21:54:32.299345 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:32.299316 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-fc8b4fcf-qkjlf_129df604-fd1b-446f-b593-0157d55a5ed8/telemeter-client/0.log" Jul 27 21:54:32.324233 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:32.324197 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-fc8b4fcf-qkjlf_129df604-fd1b-446f-b593-0157d55a5ed8/reload/0.log" Jul 27 21:54:32.344850 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:32.344824 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-fc8b4fcf-qkjlf_129df604-fd1b-446f-b593-0157d55a5ed8/kube-rbac-proxy/0.log" Jul 27 21:54:33.705977 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:33.705946 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-d2494_e066fd54-5796-438f-bdfc-784986a3713c/networking-console-plugin/0.log" Jul 27 21:54:34.216680 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:34.216651 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-j5rmn_4812c628-5f41-4e36-aaf6-8d62e4b6a443/console-operator/2.log" Jul 27 21:54:34.224246 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:34.224197 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-j5rmn_4812c628-5f41-4e36-aaf6-8d62e4b6a443/console-operator/3.log" Jul 27 21:54:34.593894 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:34.593855 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5ffbdbcc9c-sjlcr_8f4a44d7-6d72-481e-86fe-8040a3eb0379/console/0.log" Jul 27 21:54:34.636436 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:34.636403 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-fbz44_145465f3-f266-413f-8a06-54e1db6e1f8a/download-server/0.log" Jul 27 21:54:34.957384 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:34.957302 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6q2f4/perf-node-gather-daemonset-krkrk"] Jul 27 21:54:34.957769 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:34.957640 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3801fd76-3df5-4ae1-a917-62e1f490c3fe" containerName="splitter-graph-b2e96" Jul 27 21:54:34.957769 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:34.957651 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3801fd76-3df5-4ae1-a917-62e1f490c3fe" containerName="splitter-graph-b2e96" Jul 27 21:54:34.957769 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:34.957722 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3801fd76-3df5-4ae1-a917-62e1f490c3fe" containerName="splitter-graph-b2e96" Jul 27 21:54:34.960747 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:34.960727 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6q2f4/perf-node-gather-daemonset-krkrk" Jul 27 21:54:34.963005 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:34.962983 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6q2f4\"/\"kube-root-ca.crt\"" Jul 27 21:54:34.963137 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:34.963050 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6q2f4\"/\"openshift-service-ca.crt\"" Jul 27 21:54:34.963850 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:34.963834 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-6q2f4\"/\"default-dockercfg-99gfb\"" Jul 27 21:54:34.968124 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:34.967368 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c449ef8-01a4-4022-ba19-6b9f9ae65bd4-sys\") pod \"perf-node-gather-daemonset-krkrk\" (UID: \"4c449ef8-01a4-4022-ba19-6b9f9ae65bd4\") " pod="openshift-must-gather-6q2f4/perf-node-gather-daemonset-krkrk" Jul 27 21:54:34.968124 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:34.967430 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlnkl\" (UniqueName: \"kubernetes.io/projected/4c449ef8-01a4-4022-ba19-6b9f9ae65bd4-kube-api-access-wlnkl\") pod \"perf-node-gather-daemonset-krkrk\" (UID: \"4c449ef8-01a4-4022-ba19-6b9f9ae65bd4\") " pod="openshift-must-gather-6q2f4/perf-node-gather-daemonset-krkrk" Jul 27 21:54:34.968124 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:34.967528 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4c449ef8-01a4-4022-ba19-6b9f9ae65bd4-proc\") pod \"perf-node-gather-daemonset-krkrk\" (UID: \"4c449ef8-01a4-4022-ba19-6b9f9ae65bd4\") " pod="openshift-must-gather-6q2f4/perf-node-gather-daemonset-krkrk" Jul 27 21:54:34.968124 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:34.967641 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4c449ef8-01a4-4022-ba19-6b9f9ae65bd4-lib-modules\") pod \"perf-node-gather-daemonset-krkrk\" (UID: \"4c449ef8-01a4-4022-ba19-6b9f9ae65bd4\") " pod="openshift-must-gather-6q2f4/perf-node-gather-daemonset-krkrk" Jul 27 21:54:34.968124 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:34.967673 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4c449ef8-01a4-4022-ba19-6b9f9ae65bd4-podres\") pod \"perf-node-gather-daemonset-krkrk\" (UID: \"4c449ef8-01a4-4022-ba19-6b9f9ae65bd4\") " pod="openshift-must-gather-6q2f4/perf-node-gather-daemonset-krkrk" Jul 27 21:54:34.968124 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:34.967683 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6q2f4/perf-node-gather-daemonset-krkrk"] Jul 27 21:54:35.068228 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:35.068193 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4c449ef8-01a4-4022-ba19-6b9f9ae65bd4-proc\") pod \"perf-node-gather-daemonset-krkrk\" (UID: \"4c449ef8-01a4-4022-ba19-6b9f9ae65bd4\") " pod="openshift-must-gather-6q2f4/perf-node-gather-daemonset-krkrk" Jul 27 21:54:35.068415 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:35.068251 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4c449ef8-01a4-4022-ba19-6b9f9ae65bd4-lib-modules\") pod \"perf-node-gather-daemonset-krkrk\" (UID: \"4c449ef8-01a4-4022-ba19-6b9f9ae65bd4\") " pod="openshift-must-gather-6q2f4/perf-node-gather-daemonset-krkrk" Jul 27 21:54:35.068415 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:35.068272 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4c449ef8-01a4-4022-ba19-6b9f9ae65bd4-podres\") pod \"perf-node-gather-daemonset-krkrk\" (UID: \"4c449ef8-01a4-4022-ba19-6b9f9ae65bd4\") " pod="openshift-must-gather-6q2f4/perf-node-gather-daemonset-krkrk" Jul 27 21:54:35.068415 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:35.068309 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c449ef8-01a4-4022-ba19-6b9f9ae65bd4-sys\") pod \"perf-node-gather-daemonset-krkrk\" (UID: \"4c449ef8-01a4-4022-ba19-6b9f9ae65bd4\") " pod="openshift-must-gather-6q2f4/perf-node-gather-daemonset-krkrk" Jul 27 21:54:35.068415 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:35.068323 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4c449ef8-01a4-4022-ba19-6b9f9ae65bd4-proc\") pod \"perf-node-gather-daemonset-krkrk\" (UID: \"4c449ef8-01a4-4022-ba19-6b9f9ae65bd4\") " pod="openshift-must-gather-6q2f4/perf-node-gather-daemonset-krkrk" Jul 27 21:54:35.068415 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:35.068326 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlnkl\" (UniqueName: \"kubernetes.io/projected/4c449ef8-01a4-4022-ba19-6b9f9ae65bd4-kube-api-access-wlnkl\") pod \"perf-node-gather-daemonset-krkrk\" (UID: \"4c449ef8-01a4-4022-ba19-6b9f9ae65bd4\") " pod="openshift-must-gather-6q2f4/perf-node-gather-daemonset-krkrk" Jul 27 21:54:35.068415 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:35.068385 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c449ef8-01a4-4022-ba19-6b9f9ae65bd4-sys\") pod \"perf-node-gather-daemonset-krkrk\" (UID: \"4c449ef8-01a4-4022-ba19-6b9f9ae65bd4\") " pod="openshift-must-gather-6q2f4/perf-node-gather-daemonset-krkrk" Jul 27 21:54:35.068648 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:35.068453 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4c449ef8-01a4-4022-ba19-6b9f9ae65bd4-podres\") pod \"perf-node-gather-daemonset-krkrk\" (UID: \"4c449ef8-01a4-4022-ba19-6b9f9ae65bd4\") " pod="openshift-must-gather-6q2f4/perf-node-gather-daemonset-krkrk" Jul 27 21:54:35.068648 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:35.068453 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4c449ef8-01a4-4022-ba19-6b9f9ae65bd4-lib-modules\") pod \"perf-node-gather-daemonset-krkrk\" (UID: \"4c449ef8-01a4-4022-ba19-6b9f9ae65bd4\") " pod="openshift-must-gather-6q2f4/perf-node-gather-daemonset-krkrk" Jul 27 21:54:35.076753 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:35.076728 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlnkl\" (UniqueName: \"kubernetes.io/projected/4c449ef8-01a4-4022-ba19-6b9f9ae65bd4-kube-api-access-wlnkl\") pod \"perf-node-gather-daemonset-krkrk\" (UID: \"4c449ef8-01a4-4022-ba19-6b9f9ae65bd4\") " pod="openshift-must-gather-6q2f4/perf-node-gather-daemonset-krkrk" Jul 27 21:54:35.137197 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:35.137156 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-9qzp5_a4fc83fe-7a79-4461-9d98-c93af693d74f/volume-data-source-validator/0.log" Jul 27 21:54:35.271488 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:35.271455 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6q2f4/perf-node-gather-daemonset-krkrk" Jul 27 21:54:35.405733 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:35.405698 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6q2f4/perf-node-gather-daemonset-krkrk"] Jul 27 21:54:35.409076 ip-10-0-128-201 kubenswrapper[2577]: W0727 21:54:35.409051 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4c449ef8_01a4_4022_ba19_6b9f9ae65bd4.slice/crio-627623cc19162e76f8a393ee753e71bab81f21543204198e955bbe5e9893096b WatchSource:0}: Error finding container 627623cc19162e76f8a393ee753e71bab81f21543204198e955bbe5e9893096b: Status 404 returned error can't find the container with id 627623cc19162e76f8a393ee753e71bab81f21543204198e955bbe5e9893096b Jul 27 21:54:35.956621 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:35.956597 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ppf68_932ad643-a059-44a4-90f8-41016fc1e5c0/dns/0.log" Jul 27 21:54:35.976605 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:35.976555 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ppf68_932ad643-a059-44a4-90f8-41016fc1e5c0/kube-rbac-proxy/0.log" Jul 27 21:54:36.109159 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:36.109122 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6q2f4/perf-node-gather-daemonset-krkrk" event={"ID":"4c449ef8-01a4-4022-ba19-6b9f9ae65bd4","Type":"ContainerStarted","Data":"e622352731dd8e509a5c6fbcb1802c9aba4ab4393efe24f1bd4e8fa3715f507b"} Jul 27 21:54:36.109339 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:36.109181 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6q2f4/perf-node-gather-daemonset-krkrk" event={"ID":"4c449ef8-01a4-4022-ba19-6b9f9ae65bd4","Type":"ContainerStarted","Data":"627623cc19162e76f8a393ee753e71bab81f21543204198e955bbe5e9893096b"} Jul 27 21:54:36.109339 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:36.109273 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-6q2f4/perf-node-gather-daemonset-krkrk" Jul 27 21:54:36.125520 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:36.125464 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-6q2f4/perf-node-gather-daemonset-krkrk" podStartSLOduration=2.125445019 podStartE2EDuration="2.125445019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:54:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:54:36.123331642 +0000 UTC m=+3068.254947864" watchObservedRunningTime="2026-07-27 21:54:36.125445019 +0000 UTC m=+3068.257061245" Jul 27 21:54:36.133733 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:36.133710 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-npnjm_4c2ebad9-de50-4346-a64f-6cceecf44434/dns-node-resolver/0.log" Jul 27 21:54:36.637458 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:36.637422 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zt59k_e7bf9687-782f-40c9-bb73-e8b4b2c6a763/node-ca/0.log" Jul 27 21:54:37.346114 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:37.346080 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-64c559777f-75n27_86e4b5a1-4521-43b3-9bf8-c42ec92d21f8/router/0.log" Jul 27 21:54:37.657731 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:37.657647 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7gnng_597ea753-2547-4a21-9678-89da32128dcb/serve-healthcheck-canary/0.log" Jul 27 21:54:38.038697 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:38.038661 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-kwqx9_f33ee287-cfc5-436d-93cd-13801c2c522b/insights-operator/0.log" Jul 27 21:54:38.040064 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:38.040035 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-kwqx9_f33ee287-cfc5-436d-93cd-13801c2c522b/insights-operator/1.log" Jul 27 21:54:38.186029 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:38.185997 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wkkhn_2abb5699-2333-4c3f-b748-d3a227fbfb9c/kube-rbac-proxy/0.log" Jul 27 21:54:38.206531 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:38.206504 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wkkhn_2abb5699-2333-4c3f-b748-d3a227fbfb9c/exporter/0.log" Jul 27 21:54:38.227068 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:38.227037 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wkkhn_2abb5699-2333-4c3f-b748-d3a227fbfb9c/extractor/0.log" Jul 27 21:54:40.286406 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:40.286375 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-ff96d5959-8l6t7_2a3c43a5-306b-4c3d-a6de-c48ba29acead/manager/0.log" Jul 27 21:54:40.319978 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:40.319947 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-84fc959947-b2mh8_bb33cdd6-cc95-467b-9809-48a1e82797e4/manager/0.log" Jul 27 21:54:42.124456 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:42.124421 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-6q2f4/perf-node-gather-daemonset-krkrk" Jul 27 21:54:44.677998 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:44.677963 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-9rp9b_b2e7d196-079c-4bcd-aaf0-affbae8a0594/kube-storage-version-migrator-operator/1.log" Jul 27 21:54:44.679549 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:44.679526 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-9rp9b_b2e7d196-079c-4bcd-aaf0-affbae8a0594/kube-storage-version-migrator-operator/0.log" Jul 27 21:54:45.615475 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:45.615448 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5kfcg_bbf44a86-6e26-4584-b851-86bcbf8d10f9/kube-multus-additional-cni-plugins/0.log" Jul 27 21:54:45.634560 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:45.634526 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5kfcg_bbf44a86-6e26-4584-b851-86bcbf8d10f9/egress-router-binary-copy/0.log" Jul 27 21:54:45.657118 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:45.657087 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5kfcg_bbf44a86-6e26-4584-b851-86bcbf8d10f9/cni-plugins/0.log" Jul 27 21:54:45.675367 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:45.675339 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5kfcg_bbf44a86-6e26-4584-b851-86bcbf8d10f9/bond-cni-plugin/0.log" Jul 27 21:54:45.695027 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:45.695000 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5kfcg_bbf44a86-6e26-4584-b851-86bcbf8d10f9/routeoverride-cni/0.log" Jul 27 21:54:45.715242 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:45.715215 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5kfcg_bbf44a86-6e26-4584-b851-86bcbf8d10f9/whereabouts-cni-bincopy/0.log" Jul 27 21:54:45.735296 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:45.735274 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5kfcg_bbf44a86-6e26-4584-b851-86bcbf8d10f9/whereabouts-cni/0.log" Jul 27 21:54:45.959119 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:45.959089 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-tv2n5_8addb952-4638-47ca-a4fb-d04fb3441aa9/kube-multus/0.log" Jul 27 21:54:46.045216 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:46.045187 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-lvg8z_0c32154a-b41d-45c4-afa6-240a8ea24835/network-metrics-daemon/0.log" Jul 27 21:54:46.064203 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:46.064177 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-lvg8z_0c32154a-b41d-45c4-afa6-240a8ea24835/kube-rbac-proxy/0.log" Jul 27 21:54:46.997376 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:46.997327 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6mhn_1907dc14-1132-4eb9-a5ee-675b43df6e6d/ovn-controller/0.log" Jul 27 21:54:47.013978 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:47.013942 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6mhn_1907dc14-1132-4eb9-a5ee-675b43df6e6d/ovn-acl-logging/1.log" Jul 27 21:54:47.014365 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:47.014346 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6mhn_1907dc14-1132-4eb9-a5ee-675b43df6e6d/ovn-acl-logging/0.log" Jul 27 21:54:47.037201 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:47.037168 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6mhn_1907dc14-1132-4eb9-a5ee-675b43df6e6d/kube-rbac-proxy-node/0.log" Jul 27 21:54:47.063111 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:47.063080 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6mhn_1907dc14-1132-4eb9-a5ee-675b43df6e6d/kube-rbac-proxy-ovn-metrics/0.log" Jul 27 21:54:47.083995 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:47.083960 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6mhn_1907dc14-1132-4eb9-a5ee-675b43df6e6d/northd/0.log" Jul 27 21:54:47.106282 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:47.106251 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6mhn_1907dc14-1132-4eb9-a5ee-675b43df6e6d/nbdb/0.log" Jul 27 21:54:47.126731 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:47.126708 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6mhn_1907dc14-1132-4eb9-a5ee-675b43df6e6d/sbdb/0.log" Jul 27 21:54:47.195004 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:47.194967 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6mhn_1907dc14-1132-4eb9-a5ee-675b43df6e6d/ovnkube-controller/0.log" Jul 27 21:54:48.329088 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:48.329056 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-7d88cb55d8-gdgs7_ce9b0c11-99b0-4de1-bd3b-b5661465bd95/check-endpoints/0.log" Jul 27 21:54:48.371895 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:48.371866 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-lrnv5_8157bc07-d223-44de-aee6-ffa0c683a2cb/network-check-target-container/0.log" Jul 27 21:54:49.172972 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:49.172942 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-64nks_f8ef5264-95a3-4193-ae7d-f0f10a7733ab/iptables-alerter/0.log" Jul 27 21:54:49.870281 ip-10-0-128-201 kubenswrapper[2577]: I0727 21:54:49.870249 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-hq8kb_6f4b2b00-fd6c-463e-b27a-e1a925557c2f/tuned/0.log"