Jul 27 17:25:52.271299 ip-10-0-140-185 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 27 17:25:52.271312 ip-10-0-140-185 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 27 17:25:52.271321 ip-10-0-140-185 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 27 17:25:52.271673 ip-10-0-140-185 systemd[1]: Failed to start Kubernetes Kubelet. Jul 27 17:26:02.286319 ip-10-0-140-185 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 27 17:26:02.286336 ip-10-0-140-185 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 94abb0997e724c8e82a99b0aa3803461 -- Jul 27 17:28:02.431919 ip-10-0-140-185 systemd[1]: Starting Kubernetes Kubelet... Jul 27 17:28:02.843367 ip-10-0-140-185 kubenswrapper[2570]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 27 17:28:02.843367 ip-10-0-140-185 kubenswrapper[2570]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 27 17:28:02.843367 ip-10-0-140-185 kubenswrapper[2570]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 27 17:28:02.843367 ip-10-0-140-185 kubenswrapper[2570]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 27 17:28:02.843367 ip-10-0-140-185 kubenswrapper[2570]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 27 17:28:02.845060 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.844967 2570 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 27 17:28:02.848848 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.848828 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 17:28:02.848848 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.848846 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 17:28:02.848848 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.848851 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 17:28:02.848953 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.848854 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 17:28:02.848953 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.848858 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 17:28:02.848953 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.848861 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 17:28:02.848953 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.848864 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 17:28:02.848953 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.848867 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 17:28:02.848953 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.848870 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 17:28:02.848953 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.848873 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 17:28:02.848953 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.848876 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 17:28:02.848953 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.848879 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 17:28:02.848953 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.848882 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 17:28:02.848953 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.848884 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 17:28:02.848953 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.848887 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 17:28:02.848953 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.848889 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 17:28:02.848953 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.848892 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 17:28:02.848953 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.848895 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 17:28:02.848953 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.848897 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 17:28:02.848953 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.848900 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 17:28:02.848953 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.848902 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 17:28:02.848953 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.848906 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 17:28:02.848953 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.848909 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 17:28:02.849665 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.848912 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 17:28:02.849665 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.848915 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 17:28:02.849665 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.848918 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 17:28:02.849665 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.848921 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 17:28:02.849665 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.848923 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 17:28:02.849665 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.848926 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 17:28:02.849665 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.848929 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 17:28:02.849665 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.848932 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 17:28:02.849665 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.848934 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 17:28:02.849665 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.848938 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 17:28:02.849665 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.848943 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 17:28:02.849665 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.848946 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 17:28:02.849665 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.848949 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 17:28:02.849665 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.848952 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 17:28:02.849665 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.848955 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 17:28:02.849665 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.848958 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 17:28:02.849665 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.848961 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 17:28:02.849665 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.848964 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 17:28:02.849665 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.848967 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 17:28:02.850339 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.848970 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 17:28:02.850339 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.848972 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 17:28:02.850339 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.848975 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 17:28:02.850339 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.848977 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 17:28:02.850339 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.848980 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 17:28:02.850339 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.848982 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 17:28:02.850339 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.848985 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 17:28:02.850339 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.848988 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 17:28:02.850339 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.848991 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 17:28:02.850339 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.848993 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 17:28:02.850339 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.848996 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 17:28:02.850339 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.848999 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 17:28:02.850339 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.849054 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 17:28:02.850339 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.849108 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 17:28:02.850339 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.849114 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 17:28:02.850339 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.849119 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 17:28:02.850339 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.849124 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 17:28:02.850339 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.849128 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 17:28:02.850339 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.849133 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 17:28:02.850339 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.849139 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 17:28:02.850900 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.849143 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 17:28:02.850900 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.849148 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 17:28:02.850900 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.849152 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 17:28:02.850900 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.849158 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 17:28:02.850900 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.849165 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 17:28:02.850900 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.849177 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 17:28:02.850900 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.849182 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 17:28:02.850900 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.850367 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 17:28:02.850900 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.850420 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 27 17:28:02.850900 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.850594 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 17:28:02.850900 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.850601 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 17:28:02.850900 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.850604 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 17:28:02.850900 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.850612 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 17:28:02.850900 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.850615 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 17:28:02.850900 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.850618 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 17:28:02.850900 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.850622 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 17:28:02.850900 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.850625 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 17:28:02.850900 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.850628 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 17:28:02.850900 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.850631 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 17:28:02.850900 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.850634 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 17:28:02.851373 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.850636 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 17:28:02.851373 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.850641 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 17:28:02.851373 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.850646 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 17:28:02.851373 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.850649 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 17:28:02.851373 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.850652 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 17:28:02.851373 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.850656 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 17:28:02.851373 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.850658 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 17:28:02.851373 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.850662 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 17:28:02.851373 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.850665 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 17:28:02.851373 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.850668 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 17:28:02.851373 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.850671 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 17:28:02.851373 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.850674 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 17:28:02.851373 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.850676 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 17:28:02.851373 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.850679 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 17:28:02.851373 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.850682 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 17:28:02.851373 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.850685 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 17:28:02.851373 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.850687 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 17:28:02.851799 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851455 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 17:28:02.851799 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851464 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 17:28:02.851799 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851468 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 17:28:02.851799 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851471 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 17:28:02.851799 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851474 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 17:28:02.851799 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851476 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 17:28:02.851799 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851479 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 17:28:02.851799 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851481 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 17:28:02.851799 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851484 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 17:28:02.851799 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851487 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 17:28:02.851799 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851490 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 17:28:02.851799 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851492 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 17:28:02.851799 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851495 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 17:28:02.851799 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851499 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 17:28:02.851799 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851502 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 17:28:02.851799 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851505 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 17:28:02.851799 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851508 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 17:28:02.851799 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851510 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 17:28:02.851799 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851513 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 17:28:02.852262 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851515 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 17:28:02.852262 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851536 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 17:28:02.852262 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851539 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 17:28:02.852262 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851542 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 17:28:02.852262 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851545 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 17:28:02.852262 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851548 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 17:28:02.852262 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851551 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 17:28:02.852262 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851554 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 17:28:02.852262 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851556 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 17:28:02.852262 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851559 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 17:28:02.852262 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851561 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 17:28:02.852262 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851564 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 17:28:02.852262 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851566 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 17:28:02.852262 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851570 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 17:28:02.852262 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851573 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 17:28:02.852262 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851576 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 17:28:02.852262 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851579 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 17:28:02.852262 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851581 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 17:28:02.852262 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851584 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 17:28:02.852262 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851587 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 17:28:02.852784 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851590 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 17:28:02.852784 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851593 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 17:28:02.852784 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851596 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 17:28:02.852784 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851598 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 17:28:02.852784 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851601 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 17:28:02.852784 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851605 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 17:28:02.852784 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851607 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 17:28:02.852784 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851610 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 17:28:02.852784 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851613 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 17:28:02.852784 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851616 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 17:28:02.852784 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851618 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 17:28:02.852784 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851621 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 17:28:02.852784 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851623 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 17:28:02.852784 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851626 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 17:28:02.852784 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851628 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 17:28:02.852784 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851631 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 17:28:02.852784 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851633 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 17:28:02.852784 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851636 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 17:28:02.852784 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851639 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 17:28:02.852784 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851641 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 17:28:02.853277 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851644 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 17:28:02.853277 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851646 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 17:28:02.853277 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851649 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 17:28:02.853277 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851652 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 17:28:02.853277 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851655 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 27 17:28:02.853277 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851658 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 17:28:02.853277 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851661 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 17:28:02.853277 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851664 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 17:28:02.853277 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851666 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 17:28:02.853277 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851669 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 17:28:02.853277 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851671 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 17:28:02.853277 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851674 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 17:28:02.853277 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851676 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 17:28:02.853277 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851680 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 17:28:02.853277 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851684 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 17:28:02.853277 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851689 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 17:28:02.853277 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851693 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 17:28:02.853277 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851695 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 17:28:02.853277 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851699 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 17:28:02.853277 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851702 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 17:28:02.853863 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851704 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 17:28:02.853863 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851707 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 17:28:02.853863 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851709 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 17:28:02.853863 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851712 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 17:28:02.853863 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851714 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 17:28:02.853863 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851717 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 17:28:02.853863 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851719 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 17:28:02.853863 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851722 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 17:28:02.853863 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851724 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 17:28:02.853863 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851728 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 17:28:02.853863 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851731 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 17:28:02.853863 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851733 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 17:28:02.853863 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851737 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 17:28:02.853863 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851741 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 17:28:02.853863 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851743 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 17:28:02.853863 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851746 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 17:28:02.853863 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851749 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 17:28:02.853863 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851752 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 17:28:02.853863 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851755 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 17:28:02.854320 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.851758 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 17:28:02.854320 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852180 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 17:28:02.854320 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852186 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 17:28:02.854320 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852190 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 17:28:02.854320 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852193 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 17:28:02.854320 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852197 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 17:28:02.854320 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852200 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 17:28:02.854320 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852203 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 17:28:02.854320 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852206 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 17:28:02.854320 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852209 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 17:28:02.854320 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852211 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 17:28:02.854320 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852214 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 17:28:02.854320 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852216 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 17:28:02.854320 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852219 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 17:28:02.854320 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852222 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 17:28:02.854320 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852224 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 17:28:02.854320 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852227 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 17:28:02.854320 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852230 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 17:28:02.854320 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852232 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 17:28:02.854320 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852235 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 17:28:02.854849 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852238 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 17:28:02.854849 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852240 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 17:28:02.854849 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852243 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 17:28:02.854849 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852246 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 17:28:02.854849 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852249 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 17:28:02.854849 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852252 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 27 17:28:02.854849 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852255 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 17:28:02.854849 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852258 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 17:28:02.854849 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852260 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 17:28:02.854849 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852263 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 17:28:02.854849 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852266 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 17:28:02.854849 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852269 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 17:28:02.854849 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852272 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 17:28:02.854849 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852274 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 17:28:02.854849 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852277 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 17:28:02.854849 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852280 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 17:28:02.854849 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852282 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 17:28:02.854849 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852285 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 17:28:02.854849 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852288 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 17:28:02.854849 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852290 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 17:28:02.855338 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852293 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 17:28:02.855338 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852295 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 17:28:02.855338 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852298 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 17:28:02.855338 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852301 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 17:28:02.855338 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852303 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 17:28:02.855338 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852306 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 17:28:02.855338 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852308 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 17:28:02.855338 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852311 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 17:28:02.855338 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852313 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 17:28:02.855338 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852316 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 17:28:02.855338 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852318 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 17:28:02.855338 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852321 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 17:28:02.855338 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852323 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 17:28:02.855338 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852326 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 17:28:02.855338 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852329 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 17:28:02.855338 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852332 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 17:28:02.855338 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852335 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 17:28:02.855338 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852338 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 17:28:02.855338 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852340 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 17:28:02.855338 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852343 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 17:28:02.855856 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852346 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 17:28:02.855856 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852348 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 17:28:02.855856 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852351 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 17:28:02.855856 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852353 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 17:28:02.855856 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852356 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 17:28:02.855856 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852358 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 17:28:02.855856 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852361 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 17:28:02.855856 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852363 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 17:28:02.855856 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852366 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 17:28:02.855856 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852369 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 17:28:02.855856 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852371 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 17:28:02.855856 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852375 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 17:28:02.855856 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852378 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 17:28:02.855856 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852382 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 17:28:02.855856 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852385 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 17:28:02.855856 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852388 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 17:28:02.855856 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852390 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 17:28:02.855856 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852393 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 17:28:02.855856 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852395 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 17:28:02.856319 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852397 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 17:28:02.856319 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852400 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 17:28:02.856319 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852402 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 17:28:02.856319 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852405 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 17:28:02.856319 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852408 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 17:28:02.856319 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852410 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 17:28:02.856319 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852412 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 17:28:02.856319 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852415 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 17:28:02.856319 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852419 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 17:28:02.856319 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852421 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 17:28:02.856319 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852424 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 17:28:02.856319 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852426 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 17:28:02.856319 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852429 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 17:28:02.856319 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852431 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 17:28:02.856319 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852434 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 17:28:02.856319 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852436 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 17:28:02.856319 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852441 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 17:28:02.856319 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852445 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 17:28:02.856319 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852448 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 17:28:02.856807 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852454 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 17:28:02.856807 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.852467 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 17:28:02.856807 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852570 2570 flags.go:64] FLAG: --address="0.0.0.0" Jul 27 17:28:02.856807 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852579 2570 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 27 17:28:02.856807 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852587 2570 flags.go:64] FLAG: --anonymous-auth="true" Jul 27 17:28:02.856807 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852591 2570 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 27 17:28:02.856807 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852596 2570 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 27 17:28:02.856807 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852599 2570 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 27 17:28:02.856807 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852604 2570 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 27 17:28:02.856807 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852608 2570 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 27 17:28:02.856807 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852611 2570 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 27 17:28:02.856807 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852614 2570 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 27 17:28:02.856807 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852617 2570 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 27 17:28:02.856807 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852621 2570 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 27 17:28:02.856807 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852624 2570 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 27 17:28:02.856807 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852627 2570 flags.go:64] FLAG: --cgroup-root="" Jul 27 17:28:02.856807 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852630 2570 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 27 17:28:02.856807 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852634 2570 flags.go:64] FLAG: --client-ca-file="" Jul 27 17:28:02.856807 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852637 2570 flags.go:64] FLAG: --cloud-provider="external" Jul 27 17:28:02.856807 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852640 2570 flags.go:64] FLAG: --cluster-dns="[]" Jul 27 17:28:02.856807 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852645 2570 flags.go:64] FLAG: --cluster-domain="" Jul 27 17:28:02.856807 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852648 2570 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 27 17:28:02.856807 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852651 2570 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 27 17:28:02.857353 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852654 2570 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 27 17:28:02.857353 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852658 2570 flags.go:64] FLAG: --container-log-max-files="5" Jul 27 17:28:02.857353 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852667 2570 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 27 17:28:02.857353 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852670 2570 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 27 17:28:02.857353 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852673 2570 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 27 17:28:02.857353 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852677 2570 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 27 17:28:02.857353 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852680 2570 flags.go:64] FLAG: --contention-profiling="false" Jul 27 17:28:02.857353 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852683 2570 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 27 17:28:02.857353 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852686 2570 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 27 17:28:02.857353 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852689 2570 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 27 17:28:02.857353 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852692 2570 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 27 17:28:02.857353 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852696 2570 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 27 17:28:02.857353 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852699 2570 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 27 17:28:02.857353 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852702 2570 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 27 17:28:02.857353 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852705 2570 flags.go:64] FLAG: --enable-load-reader="false" Jul 27 17:28:02.857353 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852708 2570 flags.go:64] FLAG: --enable-server="true" Jul 27 17:28:02.857353 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852712 2570 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 27 17:28:02.857353 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852717 2570 flags.go:64] FLAG: --event-burst="100" Jul 27 17:28:02.857353 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852720 2570 flags.go:64] FLAG: --event-qps="50" Jul 27 17:28:02.857353 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852723 2570 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 27 17:28:02.857353 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852726 2570 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 27 17:28:02.857353 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852731 2570 flags.go:64] FLAG: --eviction-hard="" Jul 27 17:28:02.857353 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852735 2570 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 27 17:28:02.857353 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852738 2570 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 27 17:28:02.857939 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852741 2570 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 27 17:28:02.857939 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852745 2570 flags.go:64] FLAG: --eviction-soft="" Jul 27 17:28:02.857939 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852748 2570 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 27 17:28:02.857939 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852751 2570 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 27 17:28:02.857939 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852754 2570 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 27 17:28:02.857939 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852757 2570 flags.go:64] FLAG: --experimental-mounter-path="" Jul 27 17:28:02.857939 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852760 2570 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 27 17:28:02.857939 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852763 2570 flags.go:64] FLAG: --fail-swap-on="true" Jul 27 17:28:02.857939 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852766 2570 flags.go:64] FLAG: --feature-gates="" Jul 27 17:28:02.857939 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852770 2570 flags.go:64] FLAG: --file-check-frequency="20s" Jul 27 17:28:02.857939 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852773 2570 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 27 17:28:02.857939 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852776 2570 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 27 17:28:02.857939 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852779 2570 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 27 17:28:02.857939 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852782 2570 flags.go:64] FLAG: --healthz-port="10248" Jul 27 17:28:02.857939 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852785 2570 flags.go:64] FLAG: --help="false" Jul 27 17:28:02.857939 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852788 2570 flags.go:64] FLAG: --hostname-override="ip-10-0-140-185.ec2.internal" Jul 27 17:28:02.857939 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852792 2570 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 27 17:28:02.857939 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852795 2570 flags.go:64] FLAG: --http-check-frequency="20s" Jul 27 17:28:02.857939 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852798 2570 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 27 17:28:02.857939 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852801 2570 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 27 17:28:02.857939 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852805 2570 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 27 17:28:02.857939 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852808 2570 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 27 17:28:02.857939 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852810 2570 flags.go:64] FLAG: --image-service-endpoint="" Jul 27 17:28:02.858551 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852813 2570 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 27 17:28:02.858551 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852816 2570 flags.go:64] FLAG: --kube-api-burst="100" Jul 27 17:28:02.858551 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852820 2570 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 27 17:28:02.858551 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852823 2570 flags.go:64] FLAG: --kube-api-qps="50" Jul 27 17:28:02.858551 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852826 2570 flags.go:64] FLAG: --kube-reserved="" Jul 27 17:28:02.858551 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852830 2570 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 27 17:28:02.858551 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852834 2570 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 27 17:28:02.858551 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852837 2570 flags.go:64] FLAG: --kubelet-cgroups="" Jul 27 17:28:02.858551 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852840 2570 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 27 17:28:02.858551 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852842 2570 flags.go:64] FLAG: --lock-file="" Jul 27 17:28:02.858551 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852845 2570 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 27 17:28:02.858551 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852848 2570 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 27 17:28:02.858551 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852851 2570 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 27 17:28:02.858551 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852857 2570 flags.go:64] FLAG: --log-json-split-stream="false" Jul 27 17:28:02.858551 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852860 2570 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 27 17:28:02.858551 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852863 2570 flags.go:64] FLAG: --log-text-split-stream="false" Jul 27 17:28:02.858551 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852866 2570 flags.go:64] FLAG: --logging-format="text" Jul 27 17:28:02.858551 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852869 2570 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 27 17:28:02.858551 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852872 2570 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 27 17:28:02.858551 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852875 2570 flags.go:64] FLAG: --manifest-url="" Jul 27 17:28:02.858551 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852878 2570 flags.go:64] FLAG: --manifest-url-header="" Jul 27 17:28:02.858551 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852883 2570 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 27 17:28:02.858551 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852886 2570 flags.go:64] FLAG: --max-open-files="1000000" Jul 27 17:28:02.858551 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852890 2570 flags.go:64] FLAG: --max-pods="110" Jul 27 17:28:02.858551 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852893 2570 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 27 17:28:02.859154 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852896 2570 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 27 17:28:02.859154 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852899 2570 flags.go:64] FLAG: --memory-manager-policy="None" Jul 27 17:28:02.859154 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852902 2570 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 27 17:28:02.859154 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852905 2570 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 27 17:28:02.859154 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852908 2570 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 27 17:28:02.859154 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852911 2570 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 27 17:28:02.859154 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852920 2570 flags.go:64] FLAG: --node-status-max-images="50" Jul 27 17:28:02.859154 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852923 2570 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 27 17:28:02.859154 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852926 2570 flags.go:64] FLAG: --oom-score-adj="-999" Jul 27 17:28:02.859154 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852929 2570 flags.go:64] FLAG: --pod-cidr="" Jul 27 17:28:02.859154 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852935 2570 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 27 17:28:02.859154 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852940 2570 flags.go:64] FLAG: --pod-manifest-path="" Jul 27 17:28:02.859154 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852943 2570 flags.go:64] FLAG: --pod-max-pids="-1" Jul 27 17:28:02.859154 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852947 2570 flags.go:64] FLAG: --pods-per-core="0" Jul 27 17:28:02.859154 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852951 2570 flags.go:64] FLAG: --port="10250" Jul 27 17:28:02.859154 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852954 2570 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 27 17:28:02.859154 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852957 2570 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ed0d6e4ddde1f2d9" Jul 27 17:28:02.859154 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852960 2570 flags.go:64] FLAG: --qos-reserved="" Jul 27 17:28:02.859154 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852963 2570 flags.go:64] FLAG: --read-only-port="10255" Jul 27 17:28:02.859154 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852966 2570 flags.go:64] FLAG: --register-node="true" Jul 27 17:28:02.859154 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852969 2570 flags.go:64] FLAG: --register-with-taints="" Jul 27 17:28:02.859154 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852973 2570 flags.go:64] FLAG: --registry-burst="10" Jul 27 17:28:02.859154 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852976 2570 flags.go:64] FLAG: --registry-qps="5" Jul 27 17:28:02.859154 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852978 2570 flags.go:64] FLAG: --reserved-cpus="" Jul 27 17:28:02.860005 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852982 2570 flags.go:64] FLAG: --reserved-memory="" Jul 27 17:28:02.860005 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852985 2570 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 27 17:28:02.860005 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852988 2570 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 27 17:28:02.860005 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852991 2570 flags.go:64] FLAG: --rotate-certificates="false" Jul 27 17:28:02.860005 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852994 2570 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 27 17:28:02.860005 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.852997 2570 flags.go:64] FLAG: --runonce="false" Jul 27 17:28:02.860005 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.853000 2570 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 27 17:28:02.860005 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.853003 2570 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 27 17:28:02.860005 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.853006 2570 flags.go:64] FLAG: --seccomp-default="false" Jul 27 17:28:02.860005 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.853008 2570 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 27 17:28:02.860005 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.853011 2570 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 27 17:28:02.860005 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.853014 2570 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 27 17:28:02.860005 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.853017 2570 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 27 17:28:02.860005 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.853020 2570 flags.go:64] FLAG: --storage-driver-password="root" Jul 27 17:28:02.860005 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.853023 2570 flags.go:64] FLAG: --storage-driver-secure="false" Jul 27 17:28:02.860005 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.853025 2570 flags.go:64] FLAG: --storage-driver-table="stats" Jul 27 17:28:02.860005 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.853029 2570 flags.go:64] FLAG: --storage-driver-user="root" Jul 27 17:28:02.860005 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.853031 2570 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 27 17:28:02.860005 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.853035 2570 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 27 17:28:02.860005 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.853040 2570 flags.go:64] FLAG: --system-cgroups="" Jul 27 17:28:02.860005 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.853043 2570 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 27 17:28:02.860005 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.853050 2570 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 27 17:28:02.860005 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.853052 2570 flags.go:64] FLAG: --tls-cert-file="" Jul 27 17:28:02.860005 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.853055 2570 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 27 17:28:02.860005 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.853059 2570 flags.go:64] FLAG: --tls-min-version="" Jul 27 17:28:02.861110 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.853062 2570 flags.go:64] FLAG: --tls-private-key-file="" Jul 27 17:28:02.861110 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.853065 2570 flags.go:64] FLAG: --topology-manager-policy="none" Jul 27 17:28:02.861110 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.853068 2570 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 27 17:28:02.861110 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.853071 2570 flags.go:64] FLAG: --topology-manager-scope="container" Jul 27 17:28:02.861110 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.853074 2570 flags.go:64] FLAG: --v="2" Jul 27 17:28:02.861110 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.853078 2570 flags.go:64] FLAG: --version="false" Jul 27 17:28:02.861110 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.853082 2570 flags.go:64] FLAG: --vmodule="" Jul 27 17:28:02.861110 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.853086 2570 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 27 17:28:02.861110 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.853089 2570 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 27 17:28:02.861110 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853210 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 17:28:02.861110 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853215 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 17:28:02.861110 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853218 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 17:28:02.861110 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853222 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 17:28:02.861110 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853224 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 27 17:28:02.861110 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853228 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 17:28:02.861110 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853231 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 17:28:02.861110 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853233 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 17:28:02.861110 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853236 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 17:28:02.861110 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853239 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 17:28:02.861110 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853242 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 17:28:02.861110 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853244 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 17:28:02.861110 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853247 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 17:28:02.862004 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853249 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 17:28:02.862004 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853252 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 17:28:02.862004 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853255 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 17:28:02.862004 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853257 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 17:28:02.862004 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853262 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 17:28:02.862004 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853266 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 17:28:02.862004 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853272 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 17:28:02.862004 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853276 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 17:28:02.862004 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853279 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 17:28:02.862004 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853282 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 17:28:02.862004 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853285 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 17:28:02.862004 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853288 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 17:28:02.862004 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853291 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 17:28:02.862004 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853294 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 17:28:02.862004 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853297 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 17:28:02.862004 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853300 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 17:28:02.862004 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853303 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 17:28:02.862004 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853305 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 17:28:02.862004 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853308 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 17:28:02.862004 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853311 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 17:28:02.862500 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853313 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 17:28:02.862500 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853316 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 17:28:02.862500 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853319 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 17:28:02.862500 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853322 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 17:28:02.862500 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853327 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 17:28:02.862500 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853330 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 17:28:02.862500 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853333 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 17:28:02.862500 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853335 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 17:28:02.862500 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853338 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 17:28:02.862500 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853341 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 17:28:02.862500 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853343 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 17:28:02.862500 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853346 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 17:28:02.862500 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853348 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 17:28:02.862500 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853351 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 17:28:02.862500 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853353 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 17:28:02.862500 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853356 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 17:28:02.862500 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853360 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 17:28:02.862500 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853362 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 17:28:02.862500 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853367 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 17:28:02.863285 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853369 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 17:28:02.863285 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853372 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 17:28:02.863285 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853375 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 17:28:02.863285 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853377 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 17:28:02.863285 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853380 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 17:28:02.863285 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853383 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 17:28:02.863285 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853386 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 17:28:02.863285 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853388 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 17:28:02.863285 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853391 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 17:28:02.863285 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853393 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 17:28:02.863285 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853396 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 17:28:02.863285 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853398 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 17:28:02.863285 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853401 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 17:28:02.863285 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853404 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 17:28:02.863285 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853406 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 17:28:02.863285 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853409 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 17:28:02.863285 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853412 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 17:28:02.863285 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853414 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 17:28:02.863285 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853418 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 17:28:02.864081 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853422 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 17:28:02.864081 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853425 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 17:28:02.864081 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853427 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 17:28:02.864081 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853430 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 17:28:02.864081 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853433 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 17:28:02.864081 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853436 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 17:28:02.864081 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853439 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 17:28:02.864081 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853441 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 17:28:02.864081 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853444 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 17:28:02.864081 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853446 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 17:28:02.864081 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853451 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 17:28:02.864081 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853453 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 17:28:02.864081 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853457 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 17:28:02.864081 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853461 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 17:28:02.864081 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853463 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 17:28:02.864081 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853466 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 17:28:02.864081 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853468 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 17:28:02.864081 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853471 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 17:28:02.864081 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853474 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 17:28:02.864081 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853476 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 17:28:02.864693 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853479 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 17:28:02.864693 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853481 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 17:28:02.864693 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853484 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 17:28:02.864693 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853487 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 17:28:02.864693 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853489 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 17:28:02.864693 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853492 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 17:28:02.864693 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853495 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 17:28:02.864693 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.853497 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 17:28:02.864693 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.854331 2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 27 17:28:02.864693 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.861812 2570 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 27 17:28:02.864693 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.861835 2570 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 27 17:28:02.864693 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.862595 2570 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 27 17:28:02.864693 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.862610 2570 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 27 17:28:02.864693 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.862700 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 17:28:02.865263 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.862709 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 17:28:02.865263 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.862715 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 17:28:02.865263 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.862720 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 17:28:02.865263 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.862724 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 17:28:02.865263 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.862730 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 17:28:02.865263 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.862734 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 17:28:02.865263 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.862739 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 17:28:02.865263 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.862744 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 17:28:02.865263 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.862748 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 17:28:02.865263 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.862753 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 17:28:02.865263 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.862758 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 17:28:02.865263 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.862762 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 17:28:02.865263 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.862768 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 17:28:02.865263 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.862772 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 17:28:02.865263 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.862777 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 17:28:02.865263 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.862781 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 17:28:02.865263 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.862786 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 17:28:02.865263 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.862791 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 17:28:02.865263 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.862795 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 17:28:02.866194 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.862800 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 17:28:02.866194 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.862804 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 17:28:02.866194 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.862809 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 17:28:02.866194 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.862813 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 17:28:02.866194 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.862818 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 17:28:02.866194 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.862822 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 17:28:02.866194 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.862827 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 17:28:02.866194 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.862831 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 17:28:02.866194 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.862835 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 17:28:02.866194 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.862841 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 17:28:02.866194 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.862846 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 17:28:02.866194 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.862850 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 17:28:02.866194 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.862857 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 17:28:02.866194 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.862863 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 17:28:02.866194 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.862868 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 17:28:02.866194 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.862875 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 17:28:02.866194 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.862880 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 17:28:02.866194 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.862885 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 17:28:02.866194 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.862891 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 17:28:02.866784 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.862897 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 17:28:02.866784 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.862903 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 17:28:02.866784 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.862908 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 17:28:02.866784 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.862913 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 17:28:02.866784 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.862917 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 17:28:02.866784 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.862922 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 17:28:02.866784 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.862926 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 17:28:02.866784 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.862931 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 17:28:02.866784 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.862936 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 17:28:02.866784 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.862940 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 17:28:02.866784 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.862945 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 17:28:02.866784 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.862949 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 17:28:02.866784 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.862954 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 17:28:02.866784 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.862958 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 17:28:02.866784 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.862962 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 17:28:02.866784 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.862967 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 17:28:02.866784 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.862971 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 17:28:02.866784 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.862976 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 17:28:02.866784 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.862980 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 17:28:02.867287 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.862984 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 17:28:02.867287 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.862989 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 17:28:02.867287 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.862993 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 17:28:02.867287 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.862998 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 17:28:02.867287 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863003 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 17:28:02.867287 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863007 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 17:28:02.867287 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863012 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 17:28:02.867287 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863016 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 17:28:02.867287 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863021 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 17:28:02.867287 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863026 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 17:28:02.867287 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863030 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 17:28:02.867287 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863034 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 17:28:02.867287 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863038 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 17:28:02.867287 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863042 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 17:28:02.867287 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863046 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 17:28:02.867287 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863051 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 17:28:02.867287 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863055 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 17:28:02.867287 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863059 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 17:28:02.867287 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863064 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 17:28:02.867287 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863068 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 17:28:02.867797 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863072 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 17:28:02.867797 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863077 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 17:28:02.867797 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863081 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 17:28:02.867797 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863085 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 17:28:02.867797 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863089 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 17:28:02.867797 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863094 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 17:28:02.867797 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863098 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 17:28:02.867797 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863102 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 17:28:02.867797 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863107 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 17:28:02.867797 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863111 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 17:28:02.867797 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863116 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 17:28:02.867797 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863120 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 17:28:02.867797 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863124 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 17:28:02.867797 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863129 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 17:28:02.867797 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863133 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 17:28:02.867797 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863137 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 27 17:28:02.867797 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863141 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 17:28:02.867797 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863148 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 17:28:02.867797 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863152 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 17:28:02.867797 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863156 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 17:28:02.868330 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863160 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 17:28:02.868330 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.863169 2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 27 17:28:02.868330 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863353 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 17:28:02.868330 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863362 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 17:28:02.868330 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863367 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 17:28:02.868330 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863372 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 17:28:02.868330 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863376 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 17:28:02.868330 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863381 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 17:28:02.868330 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863385 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 17:28:02.868330 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863389 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 17:28:02.868330 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863393 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 17:28:02.868330 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863397 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 17:28:02.868330 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863401 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 17:28:02.868330 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863406 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 17:28:02.868330 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863411 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 17:28:02.868784 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863415 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 17:28:02.868784 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863420 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 17:28:02.868784 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863424 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 17:28:02.868784 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863429 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 17:28:02.868784 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863433 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 17:28:02.868784 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863437 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 17:28:02.868784 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863442 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 17:28:02.868784 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863446 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 17:28:02.868784 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863451 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 17:28:02.868784 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863455 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 17:28:02.868784 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863462 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 17:28:02.868784 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863468 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 17:28:02.868784 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863473 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 17:28:02.868784 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863478 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 17:28:02.868784 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863483 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 17:28:02.868784 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863493 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 17:28:02.868784 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863498 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 17:28:02.868784 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863502 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 17:28:02.868784 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863508 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 17:28:02.869246 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863512 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 17:28:02.869246 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863533 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 17:28:02.869246 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863538 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 17:28:02.869246 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863542 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 17:28:02.869246 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863546 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 17:28:02.869246 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863553 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 17:28:02.869246 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863560 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 17:28:02.869246 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863564 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 17:28:02.869246 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863569 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 17:28:02.869246 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863573 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 17:28:02.869246 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863577 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 17:28:02.869246 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863581 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 17:28:02.869246 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863586 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 17:28:02.869246 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863590 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 17:28:02.869246 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863595 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 17:28:02.869246 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863599 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 17:28:02.869246 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863604 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 17:28:02.869246 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863608 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 17:28:02.869246 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863612 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 17:28:02.869755 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863617 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 17:28:02.869755 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863622 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 17:28:02.869755 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863626 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 17:28:02.869755 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863631 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 17:28:02.869755 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863635 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 17:28:02.869755 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863640 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 17:28:02.869755 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863644 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 17:28:02.869755 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863648 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 17:28:02.869755 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863653 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 17:28:02.869755 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863659 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 17:28:02.869755 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863665 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 17:28:02.869755 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863671 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 17:28:02.869755 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863675 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 17:28:02.869755 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863680 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 17:28:02.869755 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863685 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 17:28:02.869755 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863689 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 17:28:02.869755 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863694 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 17:28:02.869755 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863698 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 17:28:02.869755 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863702 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 17:28:02.870448 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863706 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 17:28:02.870448 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863711 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 17:28:02.870448 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863715 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 17:28:02.870448 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863720 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 17:28:02.870448 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863724 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 17:28:02.870448 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863728 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 17:28:02.870448 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863733 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 17:28:02.870448 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863737 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 17:28:02.870448 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863742 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 17:28:02.870448 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863746 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 17:28:02.870448 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863750 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 17:28:02.870448 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863755 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 17:28:02.870448 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863759 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 17:28:02.870448 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863764 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 27 17:28:02.870448 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863769 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 17:28:02.870448 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863773 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 17:28:02.870448 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863777 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 17:28:02.870448 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863781 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 17:28:02.870448 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863786 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 17:28:02.870448 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863790 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 17:28:02.871345 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863794 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 17:28:02.871345 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863798 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 17:28:02.871345 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863803 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 17:28:02.871345 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863807 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 17:28:02.871345 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863812 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 17:28:02.871345 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863816 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 17:28:02.871345 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863820 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 17:28:02.871345 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863825 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 17:28:02.871345 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:02.863829 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 17:28:02.871345 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.863838 2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 27 17:28:02.871345 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.864019 2570 server.go:962] "Client rotation is on, will bootstrap in background" Jul 27 17:28:02.871345 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.866591 2570 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 27 17:28:02.871345 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.867839 2570 server.go:1019] "Starting client certificate rotation" Jul 27 17:28:02.871345 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.867952 2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 27 17:28:02.871979 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.867985 2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 27 17:28:02.871979 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.870575 2570 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 27 17:28:02.872372 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.872351 2570 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 27 17:28:02.884553 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.884498 2570 log.go:25] "Validated CRI v1 runtime API" Jul 27 17:28:02.886708 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.886691 2570 log.go:25] "Validated CRI v1 image API" Jul 27 17:28:02.887410 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.887397 2570 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 27 17:28:02.891160 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.891138 2570 fs.go:135] Filesystem UUIDs: map[4c49347a-a339-4a99-be73-7c597ac1e2eb:/dev/nvme0n1p3 6a3dd9cb-e4d4-4d0f-b08a-37402ae1f703:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Jul 27 17:28:02.891203 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.891160 2570 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 27 17:28:02.896333 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.896215 2570 manager.go:216] Machine: {Timestamp:2026-07-27 17:28:02.894930362 +0000 UTC m=+0.336145327 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100121 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2dc7b8d4fa5199c41f0fefc28d6c0f SystemUUID:ec2dc7b8-d4fa-5199-c41f-0fefc28d6c0f BootID:94abb099-7e72-4c8e-82a9-9b0aa3803461 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:26:04:f1:1e:65 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:26:04:f1:1e:65 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:06:f9:56:e5:3e:80 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 27 17:28:02.896333 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.896331 2570 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 27 17:28:02.896487 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.896471 2570 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 27 17:28:02.897282 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.897265 2570 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 27 17:28:02.899143 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.899105 2570 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 27 17:28:02.899291 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.899149 2570 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-185.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 27 17:28:02.899800 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.899788 2570 topology_manager.go:138] "Creating topology manager with none policy" Jul 27 17:28:02.899831 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.899803 2570 container_manager_linux.go:309] "Creating device plugin manager" Jul 27 17:28:02.899831 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.899810 2570 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 27 17:28:02.900511 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.900500 2570 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 27 17:28:02.901626 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.901615 2570 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 27 17:28:02.904310 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.904297 2570 state_mem.go:36] "Initialized new in-memory state store" Jul 27 17:28:02.904451 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.904441 2570 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 27 17:28:02.904495 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.904486 2570 kubelet.go:486] "Attempting to sync node with API server" Jul 27 17:28:02.904543 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.904505 2570 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 27 17:28:02.904543 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.904536 2570 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 27 17:28:02.904613 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.904548 2570 kubelet.go:398] "Adding apiserver pod source" Jul 27 17:28:02.904613 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.904561 2570 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 27 17:28:02.904953 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.904940 2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 27 17:28:02.904992 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.904960 2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 27 17:28:02.907058 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.907040 2570 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 27 17:28:02.908418 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.908399 2570 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 27 17:28:02.908476 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.908435 2570 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 27 17:28:02.909592 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.909579 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 27 17:28:02.909639 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.909601 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 27 17:28:02.909639 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.909609 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 27 17:28:02.909639 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.909614 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 27 17:28:02.909639 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.909622 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 27 17:28:02.909639 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.909628 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 27 17:28:02.909639 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.909637 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 27 17:28:02.909795 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.909643 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 27 17:28:02.909795 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.909651 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 27 17:28:02.909795 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.909657 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 27 17:28:02.909795 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.909665 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 27 17:28:02.909795 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.909670 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 27 17:28:02.910339 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.910328 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 27 17:28:02.910415 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.910340 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 27 17:28:02.913869 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.913846 2570 server.go:1268] "Started kubelet" Jul 27 17:28:02.913976 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.913940 2570 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 27 17:28:02.914130 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.914065 2570 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 27 17:28:02.914200 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.914164 2570 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 27 17:28:02.914934 ip-10-0-140-185 systemd[1]: Started Kubernetes Kubelet. Jul 27 17:28:02.915831 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:02.915755 2570 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 27 17:28:02.915922 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:02.915789 2570 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-185.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 27 17:28:02.915922 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.915872 2570 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-185.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 27 17:28:02.916448 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.916433 2570 server.go:310] "Adding debug handlers to kubelet server" Jul 27 17:28:02.919024 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.918997 2570 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 27 17:28:02.920080 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:02.920057 2570 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 27 17:28:02.920738 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.920722 2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wdjz4" Jul 27 17:28:02.920825 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:02.919882 2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-185.ec2.internal.18c635a47e989528 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-185.ec2.internal,UID:ip-10-0-140-185.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-185.ec2.internal,},FirstTimestamp:2026-07-27 17:28:02.913817896 +0000 UTC m=+0.355032861,LastTimestamp:2026-07-27 17:28:02.913817896 +0000 UTC m=+0.355032861,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-185.ec2.internal,}" Jul 27 17:28:02.925217 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.925200 2570 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 27 17:28:02.925290 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.925219 2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 27 17:28:02.925290 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:02.925275 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-185.ec2.internal\" not found" Jul 27 17:28:02.926014 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.925995 2570 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 27 17:28:02.927101 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.927077 2570 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 27 17:28:02.927203 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.927102 2570 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 27 17:28:02.927275 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.927260 2570 reconstruct.go:97] "Volume reconstruction finished" Jul 27 17:28:02.927325 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.927276 2570 reconciler.go:29] "Reconciler: start to sync state" Jul 27 17:28:02.928011 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.927582 2570 factory.go:55] Registering systemd factory Jul 27 17:28:02.928011 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:02.927691 2570 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-185.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 27 17:28:02.928011 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.927700 2570 factory.go:223] Registration of the systemd container factory successfully Jul 27 17:28:02.928011 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.927992 2570 factory.go:153] Registering CRI-O factory Jul 27 17:28:02.928011 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.928009 2570 factory.go:223] Registration of the crio container factory successfully Jul 27 17:28:02.928419 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.928068 2570 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 27 17:28:02.928419 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:02.927691 2570 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 27 17:28:02.928419 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.928096 2570 factory.go:103] Registering Raw factory Jul 27 17:28:02.928419 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.928109 2570 manager.go:1196] Started watching for new ooms in manager Jul 27 17:28:02.928890 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.928533 2570 manager.go:349] Starting recovery of all containers Jul 27 17:28:02.929324 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.929297 2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wdjz4" Jul 27 17:28:02.933955 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.933920 2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 27 17:28:02.938886 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.938869 2570 manager.go:354] Recovery completed Jul 27 17:28:02.940310 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:02.940290 2570 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Jul 27 17:28:02.943723 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.943707 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 17:28:02.945566 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.945547 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-185.ec2.internal" event="NodeHasSufficientMemory" Jul 27 17:28:02.945629 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.945579 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-185.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 17:28:02.945629 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.945590 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-185.ec2.internal" event="NodeHasSufficientPID" Jul 27 17:28:02.946172 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.946157 2570 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 27 17:28:02.946172 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.946165 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 17:28:02.946172 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.946169 2570 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 27 17:28:02.946323 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.946213 2570 state_mem.go:36] "Initialized new in-memory state store" Jul 27 17:28:02.946870 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.946856 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-185.ec2.internal" event="NodeHasSufficientMemory" Jul 27 17:28:02.946939 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.946880 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-185.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 17:28:02.946939 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.946892 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-185.ec2.internal" event="NodeHasSufficientPID" Jul 27 17:28:02.947287 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.947272 2570 policy_none.go:49] "None policy: Start" Jul 27 17:28:02.947287 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.947288 2570 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 27 17:28:02.947391 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.947299 2570 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 27 17:28:02.948201 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.948191 2570 policy_none.go:47] "Start" Jul 27 17:28:02.986397 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.986231 2570 manager.go:337] "Starting Device Plugin manager" Jul 27 17:28:02.986607 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:02.986418 2570 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 27 17:28:02.986607 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.986430 2570 server.go:87] "Starting device plugin registration server" Jul 27 17:28:02.986779 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.986767 2570 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 27 17:28:02.986827 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.986781 2570 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 27 17:28:02.986891 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.986873 2570 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 27 17:28:02.986980 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.986968 2570 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 27 17:28:02.986980 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:02.986980 2570 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 27 17:28:02.987637 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:02.987619 2570 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 27 17:28:02.987727 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:02.987672 2570 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-185.ec2.internal\" not found" Jul 27 17:28:03.087751 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.087709 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 17:28:03.089720 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.088713 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-185.ec2.internal" event="NodeHasSufficientMemory" Jul 27 17:28:03.089720 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.088741 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-185.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 17:28:03.089720 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.088751 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-185.ec2.internal" event="NodeHasSufficientPID" Jul 27 17:28:03.089720 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.088777 2570 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-185.ec2.internal" Jul 27 17:28:03.097686 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.097585 2570 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-185.ec2.internal" Jul 27 17:28:03.097686 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:03.097613 2570 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-185.ec2.internal\": node \"ip-10-0-140-185.ec2.internal\" not found" Jul 27 17:28:03.113645 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:03.113614 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-185.ec2.internal\" not found" Jul 27 17:28:03.123069 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.123041 2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 27 17:28:03.123184 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.123083 2570 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 27 17:28:03.123184 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.123104 2570 kubelet.go:2448] "Starting kubelet main sync loop" Jul 27 17:28:03.123184 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:03.123135 2570 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 27 17:28:03.126138 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.126116 2570 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 17:28:03.214001 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:03.213955 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-185.ec2.internal\" not found" Jul 27 17:28:03.224114 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.224087 2570 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-185.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-185.ec2.internal"] Jul 27 17:28:03.224221 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.224177 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 17:28:03.225160 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.225145 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-185.ec2.internal" event="NodeHasSufficientMemory" Jul 27 17:28:03.225231 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.225176 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-185.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 17:28:03.225231 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.225190 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-185.ec2.internal" event="NodeHasSufficientPID" Jul 27 17:28:03.227467 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.227455 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 17:28:03.227580 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.227566 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-185.ec2.internal" Jul 27 17:28:03.227617 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.227600 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 17:28:03.228615 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.228597 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-185.ec2.internal" event="NodeHasSufficientMemory" Jul 27 17:28:03.228615 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.228612 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-185.ec2.internal" event="NodeHasSufficientMemory" Jul 27 17:28:03.228753 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.228631 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-185.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 17:28:03.228753 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.228631 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-185.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 17:28:03.228753 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.228660 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-185.ec2.internal" event="NodeHasSufficientPID" Jul 27 17:28:03.228753 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.228641 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-185.ec2.internal" event="NodeHasSufficientPID" Jul 27 17:28:03.228964 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.228950 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/96f53b3100180b7e57940a1d1a844aea-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-185.ec2.internal\" (UID: \"96f53b3100180b7e57940a1d1a844aea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-185.ec2.internal" Jul 27 17:28:03.229017 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.228977 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/96f53b3100180b7e57940a1d1a844aea-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-185.ec2.internal\" (UID: \"96f53b3100180b7e57940a1d1a844aea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-185.ec2.internal" Jul 27 17:28:03.230889 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.230874 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-185.ec2.internal" Jul 27 17:28:03.230954 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.230897 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 17:28:03.231827 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.231810 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-185.ec2.internal" event="NodeHasSufficientMemory" Jul 27 17:28:03.231901 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.231842 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-185.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 17:28:03.231901 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.231853 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-185.ec2.internal" event="NodeHasSufficientPID" Jul 27 17:28:03.251161 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:03.251138 2570 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-185.ec2.internal\" not found" node="ip-10-0-140-185.ec2.internal" Jul 27 17:28:03.255229 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:03.255212 2570 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-185.ec2.internal\" not found" node="ip-10-0-140-185.ec2.internal" Jul 27 17:28:03.314394 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:03.314351 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-185.ec2.internal\" not found" Jul 27 17:28:03.329714 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.329688 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/96f53b3100180b7e57940a1d1a844aea-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-185.ec2.internal\" (UID: \"96f53b3100180b7e57940a1d1a844aea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-185.ec2.internal" Jul 27 17:28:03.329810 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.329725 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/92aa1923735291b61478f523669c7af8-config\") pod \"kube-apiserver-proxy-ip-10-0-140-185.ec2.internal\" (UID: \"92aa1923735291b61478f523669c7af8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-185.ec2.internal" Jul 27 17:28:03.329810 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.329752 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/96f53b3100180b7e57940a1d1a844aea-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-185.ec2.internal\" (UID: \"96f53b3100180b7e57940a1d1a844aea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-185.ec2.internal" Jul 27 17:28:03.329810 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.329786 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/96f53b3100180b7e57940a1d1a844aea-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-185.ec2.internal\" (UID: \"96f53b3100180b7e57940a1d1a844aea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-185.ec2.internal" Jul 27 17:28:03.329928 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.329815 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/96f53b3100180b7e57940a1d1a844aea-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-185.ec2.internal\" (UID: \"96f53b3100180b7e57940a1d1a844aea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-185.ec2.internal" Jul 27 17:28:03.415057 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:03.414995 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-185.ec2.internal\" not found" Jul 27 17:28:03.430288 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.430266 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/92aa1923735291b61478f523669c7af8-config\") pod \"kube-apiserver-proxy-ip-10-0-140-185.ec2.internal\" (UID: \"92aa1923735291b61478f523669c7af8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-185.ec2.internal" Jul 27 17:28:03.430354 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.430315 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/92aa1923735291b61478f523669c7af8-config\") pod \"kube-apiserver-proxy-ip-10-0-140-185.ec2.internal\" (UID: \"92aa1923735291b61478f523669c7af8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-185.ec2.internal" Jul 27 17:28:03.515436 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:03.515393 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-185.ec2.internal\" not found" Jul 27 17:28:03.553847 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.553821 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-185.ec2.internal" Jul 27 17:28:03.557830 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.557801 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-185.ec2.internal" Jul 27 17:28:03.615845 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:03.615793 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-185.ec2.internal\" not found" Jul 27 17:28:03.716238 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:03.716145 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-185.ec2.internal\" not found" Jul 27 17:28:03.766290 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.766261 2570 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 17:28:03.796253 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.796226 2570 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 17:28:03.825472 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.825444 2570 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-185.ec2.internal" Jul 27 17:28:03.837515 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.837494 2570 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 27 17:28:03.838568 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.838547 2570 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-185.ec2.internal" Jul 27 17:28:03.845364 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.845346 2570 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 27 17:28:03.867447 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.867423 2570 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 27 17:28:03.867659 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.867631 2570 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 27 17:28:03.867659 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.867632 2570 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 27 17:28:03.867659 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.867630 2570 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 27 17:28:03.905641 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.905611 2570 apiserver.go:52] "Watching apiserver" Jul 27 17:28:03.919205 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.919176 2570 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 27 17:28:03.920279 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.920259 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-d4nqw","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lnst","openshift-cluster-node-tuning-operator/tuned-w4d45","openshift-dns/node-resolver-f22lz","openshift-image-registry/node-ca-f5x5s","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-185.ec2.internal","openshift-multus/multus-4mbmr","openshift-multus/network-metrics-daemon-69zjf","kube-system/konnectivity-agent-l5bjn","kube-system/kube-apiserver-proxy-ip-10-0-140-185.ec2.internal","openshift-multus/multus-additional-cni-plugins-qp76t","openshift-network-diagnostics/network-check-target-jhzgt","openshift-network-operator/iptables-alerter-8smpx","openshift-ovn-kubernetes/ovnkube-node-nh6qf"] Jul 27 17:28:03.922704 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.922684 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d4nqw" Jul 27 17:28:03.922779 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:03.922751 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d4nqw" podUID="7209ee7f-3019-4465-8175-3317741f9b09" Jul 27 17:28:03.925108 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.924902 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lnst" Jul 27 17:28:03.925463 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.925432 2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 27 17:28:03.926956 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.926930 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 27 17:28:03.927173 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.927111 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-cz8hx\"" Jul 27 17:28:03.927336 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.927240 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-w4d45" Jul 27 17:28:03.927445 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.927276 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 27 17:28:03.927986 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.927966 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 27 17:28:03.929399 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.929382 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 27 17:28:03.929705 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.929689 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-f22lz" Jul 27 17:28:03.929795 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.929778 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-nqrvj\"" Jul 27 17:28:03.929795 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.929787 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 27 17:28:03.931291 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.931278 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-27s85\"" Jul 27 17:28:03.931341 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.931310 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 27 17:28:03.931583 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.931570 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 27 17:28:03.931726 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.931711 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3fd9dbff-b2d0-4c8b-a65a-8f9434f632c9-device-dir\") pod \"aws-ebs-csi-driver-node-7lnst\" (UID: \"3fd9dbff-b2d0-4c8b-a65a-8f9434f632c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lnst" Jul 27 17:28:03.931767 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.931736 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrddb\" (UniqueName: \"kubernetes.io/projected/3fd9dbff-b2d0-4c8b-a65a-8f9434f632c9-kube-api-access-vrddb\") pod \"aws-ebs-csi-driver-node-7lnst\" (UID: \"3fd9dbff-b2d0-4c8b-a65a-8f9434f632c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lnst" Jul 27 17:28:03.931767 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.931753 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/44c736a9-a8db-4d86-8fb1-bb5287d8c319-etc-modprobe-d\") pod \"tuned-w4d45\" (UID: \"44c736a9-a8db-4d86-8fb1-bb5287d8c319\") " pod="openshift-cluster-node-tuning-operator/tuned-w4d45" Jul 27 17:28:03.931891 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.931767 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/44c736a9-a8db-4d86-8fb1-bb5287d8c319-etc-sysconfig\") pod \"tuned-w4d45\" (UID: \"44c736a9-a8db-4d86-8fb1-bb5287d8c319\") " pod="openshift-cluster-node-tuning-operator/tuned-w4d45" Jul 27 17:28:03.931891 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.931783 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44c736a9-a8db-4d86-8fb1-bb5287d8c319-etc-kubernetes\") pod \"tuned-w4d45\" (UID: \"44c736a9-a8db-4d86-8fb1-bb5287d8c319\") " pod="openshift-cluster-node-tuning-operator/tuned-w4d45" Jul 27 17:28:03.931891 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.931806 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/44c736a9-a8db-4d86-8fb1-bb5287d8c319-etc-sysctl-d\") pod \"tuned-w4d45\" (UID: \"44c736a9-a8db-4d86-8fb1-bb5287d8c319\") " pod="openshift-cluster-node-tuning-operator/tuned-w4d45" Jul 27 17:28:03.931891 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.931827 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/44c736a9-a8db-4d86-8fb1-bb5287d8c319-etc-sysctl-conf\") pod \"tuned-w4d45\" (UID: \"44c736a9-a8db-4d86-8fb1-bb5287d8c319\") " pod="openshift-cluster-node-tuning-operator/tuned-w4d45" Jul 27 17:28:03.931891 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.931842 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3fd9dbff-b2d0-4c8b-a65a-8f9434f632c9-registration-dir\") pod \"aws-ebs-csi-driver-node-7lnst\" (UID: \"3fd9dbff-b2d0-4c8b-a65a-8f9434f632c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lnst" Jul 27 17:28:03.931891 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.931855 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3fd9dbff-b2d0-4c8b-a65a-8f9434f632c9-sys-fs\") pod \"aws-ebs-csi-driver-node-7lnst\" (UID: \"3fd9dbff-b2d0-4c8b-a65a-8f9434f632c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lnst" Jul 27 17:28:03.931891 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.931869 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/44c736a9-a8db-4d86-8fb1-bb5287d8c319-etc-systemd\") pod \"tuned-w4d45\" (UID: \"44c736a9-a8db-4d86-8fb1-bb5287d8c319\") " pod="openshift-cluster-node-tuning-operator/tuned-w4d45" Jul 27 17:28:03.931891 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.931889 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/44c736a9-a8db-4d86-8fb1-bb5287d8c319-sys\") pod \"tuned-w4d45\" (UID: \"44c736a9-a8db-4d86-8fb1-bb5287d8c319\") " pod="openshift-cluster-node-tuning-operator/tuned-w4d45" Jul 27 17:28:03.932166 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.931902 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/44c736a9-a8db-4d86-8fb1-bb5287d8c319-var-lib-kubelet\") pod \"tuned-w4d45\" (UID: \"44c736a9-a8db-4d86-8fb1-bb5287d8c319\") " pod="openshift-cluster-node-tuning-operator/tuned-w4d45" Jul 27 17:28:03.932166 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.931916 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-f5x5s" Jul 27 17:28:03.932166 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.931933 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/44c736a9-a8db-4d86-8fb1-bb5287d8c319-host\") pod \"tuned-w4d45\" (UID: \"44c736a9-a8db-4d86-8fb1-bb5287d8c319\") " pod="openshift-cluster-node-tuning-operator/tuned-w4d45" Jul 27 17:28:03.932166 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.931966 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7209ee7f-3019-4465-8175-3317741f9b09-dbus\") pod \"global-pull-secret-syncer-d4nqw\" (UID: \"7209ee7f-3019-4465-8175-3317741f9b09\") " pod="kube-system/global-pull-secret-syncer-d4nqw" Jul 27 17:28:03.932166 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.931995 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/44c736a9-a8db-4d86-8fb1-bb5287d8c319-run\") pod \"tuned-w4d45\" (UID: \"44c736a9-a8db-4d86-8fb1-bb5287d8c319\") " pod="openshift-cluster-node-tuning-operator/tuned-w4d45" Jul 27 17:28:03.932166 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.932020 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/44c736a9-a8db-4d86-8fb1-bb5287d8c319-etc-tuned\") pod \"tuned-w4d45\" (UID: \"44c736a9-a8db-4d86-8fb1-bb5287d8c319\") " pod="openshift-cluster-node-tuning-operator/tuned-w4d45" Jul 27 17:28:03.932166 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.932071 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b58jm\" (UniqueName: \"kubernetes.io/projected/44c736a9-a8db-4d86-8fb1-bb5287d8c319-kube-api-access-b58jm\") pod \"tuned-w4d45\" (UID: \"44c736a9-a8db-4d86-8fb1-bb5287d8c319\") " pod="openshift-cluster-node-tuning-operator/tuned-w4d45" Jul 27 17:28:03.932166 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.932099 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7209ee7f-3019-4465-8175-3317741f9b09-kubelet-config\") pod \"global-pull-secret-syncer-d4nqw\" (UID: \"7209ee7f-3019-4465-8175-3317741f9b09\") " pod="kube-system/global-pull-secret-syncer-d4nqw" Jul 27 17:28:03.932166 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.932116 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7209ee7f-3019-4465-8175-3317741f9b09-original-pull-secret\") pod \"global-pull-secret-syncer-d4nqw\" (UID: \"7209ee7f-3019-4465-8175-3317741f9b09\") " pod="kube-system/global-pull-secret-syncer-d4nqw" Jul 27 17:28:03.932166 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.932139 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3fd9dbff-b2d0-4c8b-a65a-8f9434f632c9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7lnst\" (UID: \"3fd9dbff-b2d0-4c8b-a65a-8f9434f632c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lnst" Jul 27 17:28:03.932166 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.932162 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3fd9dbff-b2d0-4c8b-a65a-8f9434f632c9-etc-selinux\") pod \"aws-ebs-csi-driver-node-7lnst\" (UID: \"3fd9dbff-b2d0-4c8b-a65a-8f9434f632c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lnst" Jul 27 17:28:03.932457 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.932178 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/44c736a9-a8db-4d86-8fb1-bb5287d8c319-lib-modules\") pod \"tuned-w4d45\" (UID: \"44c736a9-a8db-4d86-8fb1-bb5287d8c319\") " pod="openshift-cluster-node-tuning-operator/tuned-w4d45" Jul 27 17:28:03.932457 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.932191 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/44c736a9-a8db-4d86-8fb1-bb5287d8c319-tmp\") pod \"tuned-w4d45\" (UID: \"44c736a9-a8db-4d86-8fb1-bb5287d8c319\") " pod="openshift-cluster-node-tuning-operator/tuned-w4d45" Jul 27 17:28:03.932457 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.932204 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3fd9dbff-b2d0-4c8b-a65a-8f9434f632c9-socket-dir\") pod \"aws-ebs-csi-driver-node-7lnst\" (UID: \"3fd9dbff-b2d0-4c8b-a65a-8f9434f632c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lnst" Jul 27 17:28:03.932648 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.932616 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-26 17:23:02 +0000 UTC" deadline="2028-04-26 11:18:25.273578978 +0000 UTC" Jul 27 17:28:03.932679 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.932649 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15329h50m21.340932295s" Jul 27 17:28:03.933580 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.933566 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 27 17:28:03.933643 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.933630 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 27 17:28:03.933733 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.933720 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 27 17:28:03.933947 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.933935 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4mbmr" Jul 27 17:28:03.934165 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.934145 2570 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 27 17:28:03.934257 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.934178 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-v5lbk\"" Jul 27 17:28:03.935562 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.935548 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 27 17:28:03.935652 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.935639 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 27 17:28:03.935777 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.935759 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 27 17:28:03.935821 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.935803 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-n4wh7\"" Jul 27 17:28:03.935981 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.935968 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-69zjf" Jul 27 17:28:03.936040 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:03.936024 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-69zjf" podUID="5115219e-8201-45d1-8a3f-6db85d5e741d" Jul 27 17:28:03.936076 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.935974 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 27 17:28:03.938008 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.937992 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-l5bjn" Jul 27 17:28:03.940220 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.940205 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qp76t" Jul 27 17:28:03.940502 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.940477 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 27 17:28:03.940660 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.940645 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-259qw\"" Jul 27 17:28:03.940710 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.940689 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 27 17:28:03.941916 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.941900 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-g9vfp\"" Jul 27 17:28:03.941993 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.941964 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 27 17:28:03.942096 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.942079 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 27 17:28:03.942407 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.942391 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhzgt" Jul 27 17:28:03.942481 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:03.942444 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhzgt" podUID="ec0c6ea9-94a4-41b6-9e1f-cccf978294ab" Jul 27 17:28:03.944812 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.944797 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8smpx" Jul 27 17:28:03.946864 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.946727 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 27 17:28:03.946864 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.946768 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-mqmb8\"" Jul 27 17:28:03.946864 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.946773 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 27 17:28:03.946864 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.946825 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 27 17:28:03.947322 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.947306 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" Jul 27 17:28:03.949115 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.949095 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 27 17:28:03.949453 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.949432 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 27 17:28:03.949565 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.949550 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 27 17:28:03.949611 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.949587 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 27 17:28:03.949611 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.949598 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-w7mcf\"" Jul 27 17:28:03.949687 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.949624 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 27 17:28:03.949687 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.949590 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 27 17:28:03.959663 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.959644 2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-fd5fh" Jul 27 17:28:03.967856 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:03.967792 2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-fd5fh" Jul 27 17:28:04.027043 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.027021 2570 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 27 17:28:04.032758 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.032732 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4dbb97f8-c6bc-44e9-b468-e3577bb468af-run-openvswitch\") pod \"ovnkube-node-nh6qf\" (UID: \"4dbb97f8-c6bc-44e9-b468-e3577bb468af\") " pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" Jul 27 17:28:04.032855 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.032771 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4dbb97f8-c6bc-44e9-b468-e3577bb468af-run-ovn\") pod \"ovnkube-node-nh6qf\" (UID: \"4dbb97f8-c6bc-44e9-b468-e3577bb468af\") " pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" Jul 27 17:28:04.032855 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.032799 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4dbb97f8-c6bc-44e9-b468-e3577bb468af-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nh6qf\" (UID: \"4dbb97f8-c6bc-44e9-b468-e3577bb468af\") " pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" Jul 27 17:28:04.032855 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.032825 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4dbb97f8-c6bc-44e9-b468-e3577bb468af-env-overrides\") pod \"ovnkube-node-nh6qf\" (UID: \"4dbb97f8-c6bc-44e9-b468-e3577bb468af\") " pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" Jul 27 17:28:04.032949 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.032871 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7spf\" (UniqueName: \"kubernetes.io/projected/4dbb97f8-c6bc-44e9-b468-e3577bb468af-kube-api-access-q7spf\") pod \"ovnkube-node-nh6qf\" (UID: \"4dbb97f8-c6bc-44e9-b468-e3577bb468af\") " pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" Jul 27 17:28:04.032949 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.032901 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/44c736a9-a8db-4d86-8fb1-bb5287d8c319-etc-modprobe-d\") pod \"tuned-w4d45\" (UID: \"44c736a9-a8db-4d86-8fb1-bb5287d8c319\") " pod="openshift-cluster-node-tuning-operator/tuned-w4d45" Jul 27 17:28:04.032949 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.032924 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/44c736a9-a8db-4d86-8fb1-bb5287d8c319-etc-sysctl-d\") pod \"tuned-w4d45\" (UID: \"44c736a9-a8db-4d86-8fb1-bb5287d8c319\") " pod="openshift-cluster-node-tuning-operator/tuned-w4d45" Jul 27 17:28:04.033049 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.032948 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b9d10075-b019-4d6d-9904-bdccbc7e8ca1-multus-socket-dir-parent\") pod \"multus-4mbmr\" (UID: \"b9d10075-b019-4d6d-9904-bdccbc7e8ca1\") " pod="openshift-multus/multus-4mbmr" Jul 27 17:28:04.033049 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.032979 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b9d10075-b019-4d6d-9904-bdccbc7e8ca1-hostroot\") pod \"multus-4mbmr\" (UID: \"b9d10075-b019-4d6d-9904-bdccbc7e8ca1\") " pod="openshift-multus/multus-4mbmr" Jul 27 17:28:04.033049 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.033003 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c1bf323d-120b-45ff-9117-f8142039ccfc-os-release\") pod \"multus-additional-cni-plugins-qp76t\" (UID: \"c1bf323d-120b-45ff-9117-f8142039ccfc\") " pod="openshift-multus/multus-additional-cni-plugins-qp76t" Jul 27 17:28:04.033049 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.033024 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b9d10075-b019-4d6d-9904-bdccbc7e8ca1-system-cni-dir\") pod \"multus-4mbmr\" (UID: \"b9d10075-b019-4d6d-9904-bdccbc7e8ca1\") " pod="openshift-multus/multus-4mbmr" Jul 27 17:28:04.033049 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.033040 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/44c736a9-a8db-4d86-8fb1-bb5287d8c319-etc-modprobe-d\") pod \"tuned-w4d45\" (UID: \"44c736a9-a8db-4d86-8fb1-bb5287d8c319\") " pod="openshift-cluster-node-tuning-operator/tuned-w4d45" Jul 27 17:28:04.033269 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.033049 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b9d10075-b019-4d6d-9904-bdccbc7e8ca1-host-run-multus-certs\") pod \"multus-4mbmr\" (UID: \"b9d10075-b019-4d6d-9904-bdccbc7e8ca1\") " pod="openshift-multus/multus-4mbmr" Jul 27 17:28:04.033269 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.033050 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/44c736a9-a8db-4d86-8fb1-bb5287d8c319-etc-sysctl-d\") pod \"tuned-w4d45\" (UID: \"44c736a9-a8db-4d86-8fb1-bb5287d8c319\") " pod="openshift-cluster-node-tuning-operator/tuned-w4d45" Jul 27 17:28:04.033269 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.033091 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txgkl\" (UniqueName: \"kubernetes.io/projected/b9d10075-b019-4d6d-9904-bdccbc7e8ca1-kube-api-access-txgkl\") pod \"multus-4mbmr\" (UID: \"b9d10075-b019-4d6d-9904-bdccbc7e8ca1\") " pod="openshift-multus/multus-4mbmr" Jul 27 17:28:04.033269 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.033128 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4dbb97f8-c6bc-44e9-b468-e3577bb468af-ovnkube-config\") pod \"ovnkube-node-nh6qf\" (UID: \"4dbb97f8-c6bc-44e9-b468-e3577bb468af\") " pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" Jul 27 17:28:04.033269 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.033157 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8d2f0e3-917e-4be9-adf5-e233e4f97048-tmp-dir\") pod \"node-resolver-f22lz\" (UID: \"b8d2f0e3-917e-4be9-adf5-e233e4f97048\") " pod="openshift-dns/node-resolver-f22lz" Jul 27 17:28:04.033269 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.033189 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/44c736a9-a8db-4d86-8fb1-bb5287d8c319-etc-tuned\") pod \"tuned-w4d45\" (UID: \"44c736a9-a8db-4d86-8fb1-bb5287d8c319\") " pod="openshift-cluster-node-tuning-operator/tuned-w4d45" Jul 27 17:28:04.033269 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.033216 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b58jm\" (UniqueName: \"kubernetes.io/projected/44c736a9-a8db-4d86-8fb1-bb5287d8c319-kube-api-access-b58jm\") pod \"tuned-w4d45\" (UID: \"44c736a9-a8db-4d86-8fb1-bb5287d8c319\") " pod="openshift-cluster-node-tuning-operator/tuned-w4d45" Jul 27 17:28:04.033269 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.033243 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7209ee7f-3019-4465-8175-3317741f9b09-kubelet-config\") pod \"global-pull-secret-syncer-d4nqw\" (UID: \"7209ee7f-3019-4465-8175-3317741f9b09\") " pod="kube-system/global-pull-secret-syncer-d4nqw" Jul 27 17:28:04.033612 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.033284 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c1bf323d-120b-45ff-9117-f8142039ccfc-cnibin\") pod \"multus-additional-cni-plugins-qp76t\" (UID: \"c1bf323d-120b-45ff-9117-f8142039ccfc\") " pod="openshift-multus/multus-additional-cni-plugins-qp76t" Jul 27 17:28:04.033612 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.033310 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4dbb97f8-c6bc-44e9-b468-e3577bb468af-ovnkube-script-lib\") pod \"ovnkube-node-nh6qf\" (UID: \"4dbb97f8-c6bc-44e9-b468-e3577bb468af\") " pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" Jul 27 17:28:04.033612 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.033331 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7209ee7f-3019-4465-8175-3317741f9b09-kubelet-config\") pod \"global-pull-secret-syncer-d4nqw\" (UID: \"7209ee7f-3019-4465-8175-3317741f9b09\") " pod="kube-system/global-pull-secret-syncer-d4nqw" Jul 27 17:28:04.033612 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.033335 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/44c736a9-a8db-4d86-8fb1-bb5287d8c319-tmp\") pod \"tuned-w4d45\" (UID: \"44c736a9-a8db-4d86-8fb1-bb5287d8c319\") " pod="openshift-cluster-node-tuning-operator/tuned-w4d45" Jul 27 17:28:04.033612 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.033364 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9d10075-b019-4d6d-9904-bdccbc7e8ca1-host-var-lib-kubelet\") pod \"multus-4mbmr\" (UID: \"b9d10075-b019-4d6d-9904-bdccbc7e8ca1\") " pod="openshift-multus/multus-4mbmr" Jul 27 17:28:04.033612 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.033381 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3fd9dbff-b2d0-4c8b-a65a-8f9434f632c9-device-dir\") pod \"aws-ebs-csi-driver-node-7lnst\" (UID: \"3fd9dbff-b2d0-4c8b-a65a-8f9434f632c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lnst" Jul 27 17:28:04.033612 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.033403 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfq85\" (UniqueName: \"kubernetes.io/projected/5115219e-8201-45d1-8a3f-6db85d5e741d-kube-api-access-lfq85\") pod \"network-metrics-daemon-69zjf\" (UID: \"5115219e-8201-45d1-8a3f-6db85d5e741d\") " pod="openshift-multus/network-metrics-daemon-69zjf" Jul 27 17:28:04.033612 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.033420 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c1bf323d-120b-45ff-9117-f8142039ccfc-cni-binary-copy\") pod \"multus-additional-cni-plugins-qp76t\" (UID: \"c1bf323d-120b-45ff-9117-f8142039ccfc\") " pod="openshift-multus/multus-additional-cni-plugins-qp76t" Jul 27 17:28:04.033612 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.033434 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b9d10075-b019-4d6d-9904-bdccbc7e8ca1-host-run-netns\") pod \"multus-4mbmr\" (UID: \"b9d10075-b019-4d6d-9904-bdccbc7e8ca1\") " pod="openshift-multus/multus-4mbmr" Jul 27 17:28:04.033612 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.033437 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3fd9dbff-b2d0-4c8b-a65a-8f9434f632c9-device-dir\") pod \"aws-ebs-csi-driver-node-7lnst\" (UID: \"3fd9dbff-b2d0-4c8b-a65a-8f9434f632c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lnst" Jul 27 17:28:04.033612 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.033467 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b9d10075-b019-4d6d-9904-bdccbc7e8ca1-multus-conf-dir\") pod \"multus-4mbmr\" (UID: \"b9d10075-b019-4d6d-9904-bdccbc7e8ca1\") " pod="openshift-multus/multus-4mbmr" Jul 27 17:28:04.033612 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.033486 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/44c736a9-a8db-4d86-8fb1-bb5287d8c319-host\") pod \"tuned-w4d45\" (UID: \"44c736a9-a8db-4d86-8fb1-bb5287d8c319\") " pod="openshift-cluster-node-tuning-operator/tuned-w4d45" Jul 27 17:28:04.033612 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.033496 2570 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 27 17:28:04.033612 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.033549 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/44c736a9-a8db-4d86-8fb1-bb5287d8c319-host\") pod \"tuned-w4d45\" (UID: \"44c736a9-a8db-4d86-8fb1-bb5287d8c319\") " pod="openshift-cluster-node-tuning-operator/tuned-w4d45" Jul 27 17:28:04.033612 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.033565 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7qfl\" (UniqueName: \"kubernetes.io/projected/acba0339-7147-4770-8e0b-ae7fff7eba7e-kube-api-access-k7qfl\") pod \"node-ca-f5x5s\" (UID: \"acba0339-7147-4770-8e0b-ae7fff7eba7e\") " pod="openshift-image-registry/node-ca-f5x5s" Jul 27 17:28:04.033612 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.033597 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swcl5\" (UniqueName: \"kubernetes.io/projected/c1bf323d-120b-45ff-9117-f8142039ccfc-kube-api-access-swcl5\") pod \"multus-additional-cni-plugins-qp76t\" (UID: \"c1bf323d-120b-45ff-9117-f8142039ccfc\") " pod="openshift-multus/multus-additional-cni-plugins-qp76t" Jul 27 17:28:04.034200 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.033627 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9d10075-b019-4d6d-9904-bdccbc7e8ca1-etc-kubernetes\") pod \"multus-4mbmr\" (UID: \"b9d10075-b019-4d6d-9904-bdccbc7e8ca1\") " pod="openshift-multus/multus-4mbmr" Jul 27 17:28:04.034200 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.033653 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0e66277b-cc74-421a-84d7-d1d43ce8de57-konnectivity-ca\") pod \"konnectivity-agent-l5bjn\" (UID: \"0e66277b-cc74-421a-84d7-d1d43ce8de57\") " pod="kube-system/konnectivity-agent-l5bjn" Jul 27 17:28:04.034200 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.033676 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4dbb97f8-c6bc-44e9-b468-e3577bb468af-host-run-ovn-kubernetes\") pod \"ovnkube-node-nh6qf\" (UID: \"4dbb97f8-c6bc-44e9-b468-e3577bb468af\") " pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" Jul 27 17:28:04.034200 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.033700 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/44c736a9-a8db-4d86-8fb1-bb5287d8c319-run\") pod \"tuned-w4d45\" (UID: \"44c736a9-a8db-4d86-8fb1-bb5287d8c319\") " pod="openshift-cluster-node-tuning-operator/tuned-w4d45" Jul 27 17:28:04.034200 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.033718 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7209ee7f-3019-4465-8175-3317741f9b09-original-pull-secret\") pod \"global-pull-secret-syncer-d4nqw\" (UID: \"7209ee7f-3019-4465-8175-3317741f9b09\") " pod="kube-system/global-pull-secret-syncer-d4nqw" Jul 27 17:28:04.034200 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.033733 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/acba0339-7147-4770-8e0b-ae7fff7eba7e-serviceca\") pod \"node-ca-f5x5s\" (UID: \"acba0339-7147-4770-8e0b-ae7fff7eba7e\") " pod="openshift-image-registry/node-ca-f5x5s" Jul 27 17:28:04.034200 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.033747 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4dbb97f8-c6bc-44e9-b468-e3577bb468af-systemd-units\") pod \"ovnkube-node-nh6qf\" (UID: \"4dbb97f8-c6bc-44e9-b468-e3577bb468af\") " pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" Jul 27 17:28:04.034200 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.033761 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q5ss\" (UniqueName: \"kubernetes.io/projected/b8d2f0e3-917e-4be9-adf5-e233e4f97048-kube-api-access-5q5ss\") pod \"node-resolver-f22lz\" (UID: \"b8d2f0e3-917e-4be9-adf5-e233e4f97048\") " pod="openshift-dns/node-resolver-f22lz" Jul 27 17:28:04.034200 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.033779 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3fd9dbff-b2d0-4c8b-a65a-8f9434f632c9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7lnst\" (UID: \"3fd9dbff-b2d0-4c8b-a65a-8f9434f632c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lnst" Jul 27 17:28:04.034200 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.033821 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/44c736a9-a8db-4d86-8fb1-bb5287d8c319-run\") pod \"tuned-w4d45\" (UID: \"44c736a9-a8db-4d86-8fb1-bb5287d8c319\") " pod="openshift-cluster-node-tuning-operator/tuned-w4d45" Jul 27 17:28:04.034200 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:04.033855 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 17:28:04.034200 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.033891 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/44c736a9-a8db-4d86-8fb1-bb5287d8c319-lib-modules\") pod \"tuned-w4d45\" (UID: \"44c736a9-a8db-4d86-8fb1-bb5287d8c319\") " pod="openshift-cluster-node-tuning-operator/tuned-w4d45" Jul 27 17:28:04.034200 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:04.033924 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7209ee7f-3019-4465-8175-3317741f9b09-original-pull-secret podName:7209ee7f-3019-4465-8175-3317741f9b09 nodeName:}" failed. No retries permitted until 2026-07-27 17:28:04.533905641 +0000 UTC m=+1.975120617 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7209ee7f-3019-4465-8175-3317741f9b09-original-pull-secret") pod "global-pull-secret-syncer-d4nqw" (UID: "7209ee7f-3019-4465-8175-3317741f9b09") : object "kube-system"/"original-pull-secret" not registered Jul 27 17:28:04.034200 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.033952 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5115219e-8201-45d1-8a3f-6db85d5e741d-metrics-certs\") pod \"network-metrics-daemon-69zjf\" (UID: \"5115219e-8201-45d1-8a3f-6db85d5e741d\") " pod="openshift-multus/network-metrics-daemon-69zjf" Jul 27 17:28:04.034200 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.033969 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3fd9dbff-b2d0-4c8b-a65a-8f9434f632c9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7lnst\" (UID: \"3fd9dbff-b2d0-4c8b-a65a-8f9434f632c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lnst" Jul 27 17:28:04.034200 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.033978 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c1bf323d-120b-45ff-9117-f8142039ccfc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qp76t\" (UID: \"c1bf323d-120b-45ff-9117-f8142039ccfc\") " pod="openshift-multus/multus-additional-cni-plugins-qp76t" Jul 27 17:28:04.034826 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.033983 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/44c736a9-a8db-4d86-8fb1-bb5287d8c319-lib-modules\") pod \"tuned-w4d45\" (UID: \"44c736a9-a8db-4d86-8fb1-bb5287d8c319\") " pod="openshift-cluster-node-tuning-operator/tuned-w4d45" Jul 27 17:28:04.034826 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.034008 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4dbb97f8-c6bc-44e9-b468-e3577bb468af-var-lib-openvswitch\") pod \"ovnkube-node-nh6qf\" (UID: \"4dbb97f8-c6bc-44e9-b468-e3577bb468af\") " pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" Jul 27 17:28:04.034826 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.034029 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/44c736a9-a8db-4d86-8fb1-bb5287d8c319-etc-sysctl-conf\") pod \"tuned-w4d45\" (UID: \"44c736a9-a8db-4d86-8fb1-bb5287d8c319\") " pod="openshift-cluster-node-tuning-operator/tuned-w4d45" Jul 27 17:28:04.034826 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.034044 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/44c736a9-a8db-4d86-8fb1-bb5287d8c319-etc-systemd\") pod \"tuned-w4d45\" (UID: \"44c736a9-a8db-4d86-8fb1-bb5287d8c319\") " pod="openshift-cluster-node-tuning-operator/tuned-w4d45" Jul 27 17:28:04.034826 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.034059 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7209ee7f-3019-4465-8175-3317741f9b09-dbus\") pod \"global-pull-secret-syncer-d4nqw\" (UID: \"7209ee7f-3019-4465-8175-3317741f9b09\") " pod="kube-system/global-pull-secret-syncer-d4nqw" Jul 27 17:28:04.034826 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.034096 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c1bf323d-120b-45ff-9117-f8142039ccfc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qp76t\" (UID: \"c1bf323d-120b-45ff-9117-f8142039ccfc\") " pod="openshift-multus/multus-additional-cni-plugins-qp76t" Jul 27 17:28:04.034826 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.034113 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/44c736a9-a8db-4d86-8fb1-bb5287d8c319-etc-systemd\") pod \"tuned-w4d45\" (UID: \"44c736a9-a8db-4d86-8fb1-bb5287d8c319\") " pod="openshift-cluster-node-tuning-operator/tuned-w4d45" Jul 27 17:28:04.034826 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.034134 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9b6c93b0-6e89-41e6-accd-cbacae11cf4e-iptables-alerter-script\") pod \"iptables-alerter-8smpx\" (UID: \"9b6c93b0-6e89-41e6-accd-cbacae11cf4e\") " pod="openshift-network-operator/iptables-alerter-8smpx" Jul 27 17:28:04.034826 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.034154 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6hl8\" (UniqueName: \"kubernetes.io/projected/9b6c93b0-6e89-41e6-accd-cbacae11cf4e-kube-api-access-n6hl8\") pod \"iptables-alerter-8smpx\" (UID: \"9b6c93b0-6e89-41e6-accd-cbacae11cf4e\") " pod="openshift-network-operator/iptables-alerter-8smpx" Jul 27 17:28:04.034826 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.034166 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/44c736a9-a8db-4d86-8fb1-bb5287d8c319-etc-sysctl-conf\") pod \"tuned-w4d45\" (UID: \"44c736a9-a8db-4d86-8fb1-bb5287d8c319\") " pod="openshift-cluster-node-tuning-operator/tuned-w4d45" Jul 27 17:28:04.034826 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.034170 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b9d10075-b019-4d6d-9904-bdccbc7e8ca1-host-run-k8s-cni-cncf-io\") pod \"multus-4mbmr\" (UID: \"b9d10075-b019-4d6d-9904-bdccbc7e8ca1\") " pod="openshift-multus/multus-4mbmr" Jul 27 17:28:04.034826 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.034184 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7209ee7f-3019-4465-8175-3317741f9b09-dbus\") pod \"global-pull-secret-syncer-d4nqw\" (UID: \"7209ee7f-3019-4465-8175-3317741f9b09\") " pod="kube-system/global-pull-secret-syncer-d4nqw" Jul 27 17:28:04.034826 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.034207 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4dbb97f8-c6bc-44e9-b468-e3577bb468af-host-kubelet\") pod \"ovnkube-node-nh6qf\" (UID: \"4dbb97f8-c6bc-44e9-b468-e3577bb468af\") " pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" Jul 27 17:28:04.034826 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.034231 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3fd9dbff-b2d0-4c8b-a65a-8f9434f632c9-registration-dir\") pod \"aws-ebs-csi-driver-node-7lnst\" (UID: \"3fd9dbff-b2d0-4c8b-a65a-8f9434f632c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lnst" Jul 27 17:28:04.034826 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.034246 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/44c736a9-a8db-4d86-8fb1-bb5287d8c319-sys\") pod \"tuned-w4d45\" (UID: \"44c736a9-a8db-4d86-8fb1-bb5287d8c319\") " pod="openshift-cluster-node-tuning-operator/tuned-w4d45" Jul 27 17:28:04.034826 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.034262 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c1bf323d-120b-45ff-9117-f8142039ccfc-system-cni-dir\") pod \"multus-additional-cni-plugins-qp76t\" (UID: \"c1bf323d-120b-45ff-9117-f8142039ccfc\") " pod="openshift-multus/multus-additional-cni-plugins-qp76t" Jul 27 17:28:04.034826 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.034294 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3fd9dbff-b2d0-4c8b-a65a-8f9434f632c9-registration-dir\") pod \"aws-ebs-csi-driver-node-7lnst\" (UID: \"3fd9dbff-b2d0-4c8b-a65a-8f9434f632c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lnst" Jul 27 17:28:04.035613 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.034300 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/44c736a9-a8db-4d86-8fb1-bb5287d8c319-sys\") pod \"tuned-w4d45\" (UID: \"44c736a9-a8db-4d86-8fb1-bb5287d8c319\") " pod="openshift-cluster-node-tuning-operator/tuned-w4d45" Jul 27 17:28:04.035613 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.034331 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b9d10075-b019-4d6d-9904-bdccbc7e8ca1-host-var-lib-cni-multus\") pod \"multus-4mbmr\" (UID: \"b9d10075-b019-4d6d-9904-bdccbc7e8ca1\") " pod="openshift-multus/multus-4mbmr" Jul 27 17:28:04.035613 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.034355 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0e66277b-cc74-421a-84d7-d1d43ce8de57-agent-certs\") pod \"konnectivity-agent-l5bjn\" (UID: \"0e66277b-cc74-421a-84d7-d1d43ce8de57\") " pod="kube-system/konnectivity-agent-l5bjn" Jul 27 17:28:04.035613 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.034376 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p4xw\" (UniqueName: \"kubernetes.io/projected/ec0c6ea9-94a4-41b6-9e1f-cccf978294ab-kube-api-access-5p4xw\") pod \"network-check-target-jhzgt\" (UID: \"ec0c6ea9-94a4-41b6-9e1f-cccf978294ab\") " pod="openshift-network-diagnostics/network-check-target-jhzgt" Jul 27 17:28:04.035613 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.034400 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4dbb97f8-c6bc-44e9-b468-e3577bb468af-host-slash\") pod \"ovnkube-node-nh6qf\" (UID: \"4dbb97f8-c6bc-44e9-b468-e3577bb468af\") " pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" Jul 27 17:28:04.035613 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.034421 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4dbb97f8-c6bc-44e9-b468-e3577bb468af-log-socket\") pod \"ovnkube-node-nh6qf\" (UID: \"4dbb97f8-c6bc-44e9-b468-e3577bb468af\") " pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" Jul 27 17:28:04.035613 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.034443 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b9d10075-b019-4d6d-9904-bdccbc7e8ca1-host-var-lib-cni-bin\") pod \"multus-4mbmr\" (UID: \"b9d10075-b019-4d6d-9904-bdccbc7e8ca1\") " pod="openshift-multus/multus-4mbmr" Jul 27 17:28:04.035613 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.034465 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4dbb97f8-c6bc-44e9-b468-e3577bb468af-etc-openvswitch\") pod \"ovnkube-node-nh6qf\" (UID: \"4dbb97f8-c6bc-44e9-b468-e3577bb468af\") " pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" Jul 27 17:28:04.035613 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.034489 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4dbb97f8-c6bc-44e9-b468-e3577bb468af-ovn-node-metrics-cert\") pod \"ovnkube-node-nh6qf\" (UID: \"4dbb97f8-c6bc-44e9-b468-e3577bb468af\") " pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" Jul 27 17:28:04.035613 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.034513 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3fd9dbff-b2d0-4c8b-a65a-8f9434f632c9-etc-selinux\") pod \"aws-ebs-csi-driver-node-7lnst\" (UID: \"3fd9dbff-b2d0-4c8b-a65a-8f9434f632c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lnst" Jul 27 17:28:04.035613 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.034613 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4dbb97f8-c6bc-44e9-b468-e3577bb468af-host-run-netns\") pod \"ovnkube-node-nh6qf\" (UID: \"4dbb97f8-c6bc-44e9-b468-e3577bb468af\") " pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" Jul 27 17:28:04.035613 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.034632 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4dbb97f8-c6bc-44e9-b468-e3577bb468af-host-cni-netd\") pod \"ovnkube-node-nh6qf\" (UID: \"4dbb97f8-c6bc-44e9-b468-e3577bb468af\") " pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" Jul 27 17:28:04.035613 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.034655 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b8d2f0e3-917e-4be9-adf5-e233e4f97048-hosts-file\") pod \"node-resolver-f22lz\" (UID: \"b8d2f0e3-917e-4be9-adf5-e233e4f97048\") " pod="openshift-dns/node-resolver-f22lz" Jul 27 17:28:04.035613 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.034659 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3fd9dbff-b2d0-4c8b-a65a-8f9434f632c9-etc-selinux\") pod \"aws-ebs-csi-driver-node-7lnst\" (UID: \"3fd9dbff-b2d0-4c8b-a65a-8f9434f632c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lnst" Jul 27 17:28:04.035613 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.034674 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b9d10075-b019-4d6d-9904-bdccbc7e8ca1-os-release\") pod \"multus-4mbmr\" (UID: \"b9d10075-b019-4d6d-9904-bdccbc7e8ca1\") " pod="openshift-multus/multus-4mbmr" Jul 27 17:28:04.035613 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.034697 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3fd9dbff-b2d0-4c8b-a65a-8f9434f632c9-socket-dir\") pod \"aws-ebs-csi-driver-node-7lnst\" (UID: \"3fd9dbff-b2d0-4c8b-a65a-8f9434f632c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lnst" Jul 27 17:28:04.035613 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.034734 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrddb\" (UniqueName: \"kubernetes.io/projected/3fd9dbff-b2d0-4c8b-a65a-8f9434f632c9-kube-api-access-vrddb\") pod \"aws-ebs-csi-driver-node-7lnst\" (UID: \"3fd9dbff-b2d0-4c8b-a65a-8f9434f632c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lnst" Jul 27 17:28:04.036072 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.034760 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/44c736a9-a8db-4d86-8fb1-bb5287d8c319-etc-sysconfig\") pod \"tuned-w4d45\" (UID: \"44c736a9-a8db-4d86-8fb1-bb5287d8c319\") " pod="openshift-cluster-node-tuning-operator/tuned-w4d45" Jul 27 17:28:04.036072 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.034794 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44c736a9-a8db-4d86-8fb1-bb5287d8c319-etc-kubernetes\") pod \"tuned-w4d45\" (UID: \"44c736a9-a8db-4d86-8fb1-bb5287d8c319\") " pod="openshift-cluster-node-tuning-operator/tuned-w4d45" Jul 27 17:28:04.036072 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.034883 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3fd9dbff-b2d0-4c8b-a65a-8f9434f632c9-socket-dir\") pod \"aws-ebs-csi-driver-node-7lnst\" (UID: \"3fd9dbff-b2d0-4c8b-a65a-8f9434f632c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lnst" Jul 27 17:28:04.036072 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.034905 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/44c736a9-a8db-4d86-8fb1-bb5287d8c319-etc-sysconfig\") pod \"tuned-w4d45\" (UID: \"44c736a9-a8db-4d86-8fb1-bb5287d8c319\") " pod="openshift-cluster-node-tuning-operator/tuned-w4d45" Jul 27 17:28:04.036072 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.034936 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/acba0339-7147-4770-8e0b-ae7fff7eba7e-host\") pod \"node-ca-f5x5s\" (UID: \"acba0339-7147-4770-8e0b-ae7fff7eba7e\") " pod="openshift-image-registry/node-ca-f5x5s" Jul 27 17:28:04.036072 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.035071 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b9d10075-b019-4d6d-9904-bdccbc7e8ca1-cnibin\") pod \"multus-4mbmr\" (UID: \"b9d10075-b019-4d6d-9904-bdccbc7e8ca1\") " pod="openshift-multus/multus-4mbmr" Jul 27 17:28:04.036072 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.035097 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4dbb97f8-c6bc-44e9-b468-e3577bb468af-run-systemd\") pod \"ovnkube-node-nh6qf\" (UID: \"4dbb97f8-c6bc-44e9-b468-e3577bb468af\") " pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" Jul 27 17:28:04.036072 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.035122 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3fd9dbff-b2d0-4c8b-a65a-8f9434f632c9-sys-fs\") pod \"aws-ebs-csi-driver-node-7lnst\" (UID: \"3fd9dbff-b2d0-4c8b-a65a-8f9434f632c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lnst" Jul 27 17:28:04.036072 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.035200 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44c736a9-a8db-4d86-8fb1-bb5287d8c319-etc-kubernetes\") pod \"tuned-w4d45\" (UID: \"44c736a9-a8db-4d86-8fb1-bb5287d8c319\") " pod="openshift-cluster-node-tuning-operator/tuned-w4d45" Jul 27 17:28:04.036072 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.035201 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/44c736a9-a8db-4d86-8fb1-bb5287d8c319-var-lib-kubelet\") pod \"tuned-w4d45\" (UID: \"44c736a9-a8db-4d86-8fb1-bb5287d8c319\") " pod="openshift-cluster-node-tuning-operator/tuned-w4d45" Jul 27 17:28:04.036072 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.035223 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3fd9dbff-b2d0-4c8b-a65a-8f9434f632c9-sys-fs\") pod \"aws-ebs-csi-driver-node-7lnst\" (UID: \"3fd9dbff-b2d0-4c8b-a65a-8f9434f632c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lnst" Jul 27 17:28:04.036072 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.035237 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9b6c93b0-6e89-41e6-accd-cbacae11cf4e-host-slash\") pod \"iptables-alerter-8smpx\" (UID: \"9b6c93b0-6e89-41e6-accd-cbacae11cf4e\") " pod="openshift-network-operator/iptables-alerter-8smpx" Jul 27 17:28:04.036072 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.035247 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/44c736a9-a8db-4d86-8fb1-bb5287d8c319-var-lib-kubelet\") pod \"tuned-w4d45\" (UID: \"44c736a9-a8db-4d86-8fb1-bb5287d8c319\") " pod="openshift-cluster-node-tuning-operator/tuned-w4d45" Jul 27 17:28:04.036072 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.035261 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b9d10075-b019-4d6d-9904-bdccbc7e8ca1-multus-cni-dir\") pod \"multus-4mbmr\" (UID: \"b9d10075-b019-4d6d-9904-bdccbc7e8ca1\") " pod="openshift-multus/multus-4mbmr" Jul 27 17:28:04.036072 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.035285 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b9d10075-b019-4d6d-9904-bdccbc7e8ca1-multus-daemon-config\") pod \"multus-4mbmr\" (UID: \"b9d10075-b019-4d6d-9904-bdccbc7e8ca1\") " pod="openshift-multus/multus-4mbmr" Jul 27 17:28:04.036072 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.035308 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4dbb97f8-c6bc-44e9-b468-e3577bb468af-node-log\") pod \"ovnkube-node-nh6qf\" (UID: \"4dbb97f8-c6bc-44e9-b468-e3577bb468af\") " pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" Jul 27 17:28:04.036072 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.035330 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4dbb97f8-c6bc-44e9-b468-e3577bb468af-host-cni-bin\") pod \"ovnkube-node-nh6qf\" (UID: \"4dbb97f8-c6bc-44e9-b468-e3577bb468af\") " pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" Jul 27 17:28:04.036683 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.035367 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c1bf323d-120b-45ff-9117-f8142039ccfc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qp76t\" (UID: \"c1bf323d-120b-45ff-9117-f8142039ccfc\") " pod="openshift-multus/multus-additional-cni-plugins-qp76t" Jul 27 17:28:04.036683 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.035391 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b9d10075-b019-4d6d-9904-bdccbc7e8ca1-cni-binary-copy\") pod \"multus-4mbmr\" (UID: \"b9d10075-b019-4d6d-9904-bdccbc7e8ca1\") " pod="openshift-multus/multus-4mbmr" Jul 27 17:28:04.036683 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.036620 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/44c736a9-a8db-4d86-8fb1-bb5287d8c319-etc-tuned\") pod \"tuned-w4d45\" (UID: \"44c736a9-a8db-4d86-8fb1-bb5287d8c319\") " pod="openshift-cluster-node-tuning-operator/tuned-w4d45" Jul 27 17:28:04.036795 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.036690 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/44c736a9-a8db-4d86-8fb1-bb5287d8c319-tmp\") pod \"tuned-w4d45\" (UID: \"44c736a9-a8db-4d86-8fb1-bb5287d8c319\") " pod="openshift-cluster-node-tuning-operator/tuned-w4d45" Jul 27 17:28:04.042765 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.042744 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b58jm\" (UniqueName: \"kubernetes.io/projected/44c736a9-a8db-4d86-8fb1-bb5287d8c319-kube-api-access-b58jm\") pod \"tuned-w4d45\" (UID: \"44c736a9-a8db-4d86-8fb1-bb5287d8c319\") " pod="openshift-cluster-node-tuning-operator/tuned-w4d45" Jul 27 17:28:04.043231 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.043215 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrddb\" (UniqueName: \"kubernetes.io/projected/3fd9dbff-b2d0-4c8b-a65a-8f9434f632c9-kube-api-access-vrddb\") pod \"aws-ebs-csi-driver-node-7lnst\" (UID: \"3fd9dbff-b2d0-4c8b-a65a-8f9434f632c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lnst" Jul 27 17:28:04.094138 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.094104 2570 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 17:28:04.135688 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.135654 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4dbb97f8-c6bc-44e9-b468-e3577bb468af-host-run-netns\") pod \"ovnkube-node-nh6qf\" (UID: \"4dbb97f8-c6bc-44e9-b468-e3577bb468af\") " pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" Jul 27 17:28:04.135688 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.135689 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4dbb97f8-c6bc-44e9-b468-e3577bb468af-host-cni-netd\") pod \"ovnkube-node-nh6qf\" (UID: \"4dbb97f8-c6bc-44e9-b468-e3577bb468af\") " pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" Jul 27 17:28:04.135950 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.135705 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b8d2f0e3-917e-4be9-adf5-e233e4f97048-hosts-file\") pod \"node-resolver-f22lz\" (UID: \"b8d2f0e3-917e-4be9-adf5-e233e4f97048\") " pod="openshift-dns/node-resolver-f22lz" Jul 27 17:28:04.135950 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.135721 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b9d10075-b019-4d6d-9904-bdccbc7e8ca1-os-release\") pod \"multus-4mbmr\" (UID: \"b9d10075-b019-4d6d-9904-bdccbc7e8ca1\") " pod="openshift-multus/multus-4mbmr" Jul 27 17:28:04.135950 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.135767 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4dbb97f8-c6bc-44e9-b468-e3577bb468af-host-run-netns\") pod \"ovnkube-node-nh6qf\" (UID: \"4dbb97f8-c6bc-44e9-b468-e3577bb468af\") " pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" Jul 27 17:28:04.135950 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.135784 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4dbb97f8-c6bc-44e9-b468-e3577bb468af-host-cni-netd\") pod \"ovnkube-node-nh6qf\" (UID: \"4dbb97f8-c6bc-44e9-b468-e3577bb468af\") " pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" Jul 27 17:28:04.135950 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.135781 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b9d10075-b019-4d6d-9904-bdccbc7e8ca1-os-release\") pod \"multus-4mbmr\" (UID: \"b9d10075-b019-4d6d-9904-bdccbc7e8ca1\") " pod="openshift-multus/multus-4mbmr" Jul 27 17:28:04.135950 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.135806 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b8d2f0e3-917e-4be9-adf5-e233e4f97048-hosts-file\") pod \"node-resolver-f22lz\" (UID: \"b8d2f0e3-917e-4be9-adf5-e233e4f97048\") " pod="openshift-dns/node-resolver-f22lz" Jul 27 17:28:04.135950 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.135830 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/acba0339-7147-4770-8e0b-ae7fff7eba7e-host\") pod \"node-ca-f5x5s\" (UID: \"acba0339-7147-4770-8e0b-ae7fff7eba7e\") " pod="openshift-image-registry/node-ca-f5x5s" Jul 27 17:28:04.135950 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.135860 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b9d10075-b019-4d6d-9904-bdccbc7e8ca1-cnibin\") pod \"multus-4mbmr\" (UID: \"b9d10075-b019-4d6d-9904-bdccbc7e8ca1\") " pod="openshift-multus/multus-4mbmr" Jul 27 17:28:04.135950 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.135882 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4dbb97f8-c6bc-44e9-b468-e3577bb468af-run-systemd\") pod \"ovnkube-node-nh6qf\" (UID: \"4dbb97f8-c6bc-44e9-b468-e3577bb468af\") " pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" Jul 27 17:28:04.135950 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.135886 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/acba0339-7147-4770-8e0b-ae7fff7eba7e-host\") pod \"node-ca-f5x5s\" (UID: \"acba0339-7147-4770-8e0b-ae7fff7eba7e\") " pod="openshift-image-registry/node-ca-f5x5s" Jul 27 17:28:04.135950 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.135909 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9b6c93b0-6e89-41e6-accd-cbacae11cf4e-host-slash\") pod \"iptables-alerter-8smpx\" (UID: \"9b6c93b0-6e89-41e6-accd-cbacae11cf4e\") " pod="openshift-network-operator/iptables-alerter-8smpx" Jul 27 17:28:04.135950 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.135935 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b9d10075-b019-4d6d-9904-bdccbc7e8ca1-multus-cni-dir\") pod \"multus-4mbmr\" (UID: \"b9d10075-b019-4d6d-9904-bdccbc7e8ca1\") " pod="openshift-multus/multus-4mbmr" Jul 27 17:28:04.135950 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.135942 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4dbb97f8-c6bc-44e9-b468-e3577bb468af-run-systemd\") pod \"ovnkube-node-nh6qf\" (UID: \"4dbb97f8-c6bc-44e9-b468-e3577bb468af\") " pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" Jul 27 17:28:04.135950 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.135957 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b9d10075-b019-4d6d-9904-bdccbc7e8ca1-multus-daemon-config\") pod \"multus-4mbmr\" (UID: \"b9d10075-b019-4d6d-9904-bdccbc7e8ca1\") " pod="openshift-multus/multus-4mbmr" Jul 27 17:28:04.136567 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.135967 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9b6c93b0-6e89-41e6-accd-cbacae11cf4e-host-slash\") pod \"iptables-alerter-8smpx\" (UID: \"9b6c93b0-6e89-41e6-accd-cbacae11cf4e\") " pod="openshift-network-operator/iptables-alerter-8smpx" Jul 27 17:28:04.136567 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.135979 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4dbb97f8-c6bc-44e9-b468-e3577bb468af-node-log\") pod \"ovnkube-node-nh6qf\" (UID: \"4dbb97f8-c6bc-44e9-b468-e3577bb468af\") " pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" Jul 27 17:28:04.136567 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.135935 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b9d10075-b019-4d6d-9904-bdccbc7e8ca1-cnibin\") pod \"multus-4mbmr\" (UID: \"b9d10075-b019-4d6d-9904-bdccbc7e8ca1\") " pod="openshift-multus/multus-4mbmr" Jul 27 17:28:04.136567 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.135998 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b9d10075-b019-4d6d-9904-bdccbc7e8ca1-multus-cni-dir\") pod \"multus-4mbmr\" (UID: \"b9d10075-b019-4d6d-9904-bdccbc7e8ca1\") " pod="openshift-multus/multus-4mbmr" Jul 27 17:28:04.136567 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.136022 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4dbb97f8-c6bc-44e9-b468-e3577bb468af-node-log\") pod \"ovnkube-node-nh6qf\" (UID: \"4dbb97f8-c6bc-44e9-b468-e3577bb468af\") " pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" Jul 27 17:28:04.136567 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.136061 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4dbb97f8-c6bc-44e9-b468-e3577bb468af-host-cni-bin\") pod \"ovnkube-node-nh6qf\" (UID: \"4dbb97f8-c6bc-44e9-b468-e3577bb468af\") " pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" Jul 27 17:28:04.136567 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.136087 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c1bf323d-120b-45ff-9117-f8142039ccfc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qp76t\" (UID: \"c1bf323d-120b-45ff-9117-f8142039ccfc\") " pod="openshift-multus/multus-additional-cni-plugins-qp76t" Jul 27 17:28:04.136567 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.136104 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b9d10075-b019-4d6d-9904-bdccbc7e8ca1-cni-binary-copy\") pod \"multus-4mbmr\" (UID: \"b9d10075-b019-4d6d-9904-bdccbc7e8ca1\") " pod="openshift-multus/multus-4mbmr" Jul 27 17:28:04.136567 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.136121 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4dbb97f8-c6bc-44e9-b468-e3577bb468af-run-openvswitch\") pod \"ovnkube-node-nh6qf\" (UID: \"4dbb97f8-c6bc-44e9-b468-e3577bb468af\") " pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" Jul 27 17:28:04.136567 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.136137 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4dbb97f8-c6bc-44e9-b468-e3577bb468af-run-ovn\") pod \"ovnkube-node-nh6qf\" (UID: \"4dbb97f8-c6bc-44e9-b468-e3577bb468af\") " pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" Jul 27 17:28:04.136567 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.136154 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4dbb97f8-c6bc-44e9-b468-e3577bb468af-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nh6qf\" (UID: \"4dbb97f8-c6bc-44e9-b468-e3577bb468af\") " pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" Jul 27 17:28:04.136567 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.136176 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4dbb97f8-c6bc-44e9-b468-e3577bb468af-env-overrides\") pod \"ovnkube-node-nh6qf\" (UID: \"4dbb97f8-c6bc-44e9-b468-e3577bb468af\") " pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" Jul 27 17:28:04.136567 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.136178 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4dbb97f8-c6bc-44e9-b468-e3577bb468af-host-cni-bin\") pod \"ovnkube-node-nh6qf\" (UID: \"4dbb97f8-c6bc-44e9-b468-e3577bb468af\") " pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" Jul 27 17:28:04.136567 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.136201 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4dbb97f8-c6bc-44e9-b468-e3577bb468af-run-openvswitch\") pod \"ovnkube-node-nh6qf\" (UID: \"4dbb97f8-c6bc-44e9-b468-e3577bb468af\") " pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" Jul 27 17:28:04.136567 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.136208 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7spf\" (UniqueName: \"kubernetes.io/projected/4dbb97f8-c6bc-44e9-b468-e3577bb468af-kube-api-access-q7spf\") pod \"ovnkube-node-nh6qf\" (UID: \"4dbb97f8-c6bc-44e9-b468-e3577bb468af\") " pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" Jul 27 17:28:04.136567 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.136236 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b9d10075-b019-4d6d-9904-bdccbc7e8ca1-multus-socket-dir-parent\") pod \"multus-4mbmr\" (UID: \"b9d10075-b019-4d6d-9904-bdccbc7e8ca1\") " pod="openshift-multus/multus-4mbmr" Jul 27 17:28:04.136567 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.136239 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4dbb97f8-c6bc-44e9-b468-e3577bb468af-run-ovn\") pod \"ovnkube-node-nh6qf\" (UID: \"4dbb97f8-c6bc-44e9-b468-e3577bb468af\") " pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" Jul 27 17:28:04.136567 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.136260 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b9d10075-b019-4d6d-9904-bdccbc7e8ca1-hostroot\") pod \"multus-4mbmr\" (UID: \"b9d10075-b019-4d6d-9904-bdccbc7e8ca1\") " pod="openshift-multus/multus-4mbmr" Jul 27 17:28:04.137621 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.136260 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4dbb97f8-c6bc-44e9-b468-e3577bb468af-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nh6qf\" (UID: \"4dbb97f8-c6bc-44e9-b468-e3577bb468af\") " pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" Jul 27 17:28:04.137621 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.136278 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c1bf323d-120b-45ff-9117-f8142039ccfc-os-release\") pod \"multus-additional-cni-plugins-qp76t\" (UID: \"c1bf323d-120b-45ff-9117-f8142039ccfc\") " pod="openshift-multus/multus-additional-cni-plugins-qp76t" Jul 27 17:28:04.137621 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.136295 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b9d10075-b019-4d6d-9904-bdccbc7e8ca1-system-cni-dir\") pod \"multus-4mbmr\" (UID: \"b9d10075-b019-4d6d-9904-bdccbc7e8ca1\") " pod="openshift-multus/multus-4mbmr" Jul 27 17:28:04.137621 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.136318 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b9d10075-b019-4d6d-9904-bdccbc7e8ca1-host-run-multus-certs\") pod \"multus-4mbmr\" (UID: \"b9d10075-b019-4d6d-9904-bdccbc7e8ca1\") " pod="openshift-multus/multus-4mbmr" Jul 27 17:28:04.137621 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.136329 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b9d10075-b019-4d6d-9904-bdccbc7e8ca1-multus-socket-dir-parent\") pod \"multus-4mbmr\" (UID: \"b9d10075-b019-4d6d-9904-bdccbc7e8ca1\") " pod="openshift-multus/multus-4mbmr" Jul 27 17:28:04.137621 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.136340 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txgkl\" (UniqueName: \"kubernetes.io/projected/b9d10075-b019-4d6d-9904-bdccbc7e8ca1-kube-api-access-txgkl\") pod \"multus-4mbmr\" (UID: \"b9d10075-b019-4d6d-9904-bdccbc7e8ca1\") " pod="openshift-multus/multus-4mbmr" Jul 27 17:28:04.137621 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.136389 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b9d10075-b019-4d6d-9904-bdccbc7e8ca1-system-cni-dir\") pod \"multus-4mbmr\" (UID: \"b9d10075-b019-4d6d-9904-bdccbc7e8ca1\") " pod="openshift-multus/multus-4mbmr" Jul 27 17:28:04.137621 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.136419 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4dbb97f8-c6bc-44e9-b468-e3577bb468af-ovnkube-config\") pod \"ovnkube-node-nh6qf\" (UID: \"4dbb97f8-c6bc-44e9-b468-e3577bb468af\") " pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" Jul 27 17:28:04.137621 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.136443 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8d2f0e3-917e-4be9-adf5-e233e4f97048-tmp-dir\") pod \"node-resolver-f22lz\" (UID: \"b8d2f0e3-917e-4be9-adf5-e233e4f97048\") " pod="openshift-dns/node-resolver-f22lz" Jul 27 17:28:04.137621 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.136472 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c1bf323d-120b-45ff-9117-f8142039ccfc-cnibin\") pod \"multus-additional-cni-plugins-qp76t\" (UID: \"c1bf323d-120b-45ff-9117-f8142039ccfc\") " pod="openshift-multus/multus-additional-cni-plugins-qp76t" Jul 27 17:28:04.137621 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.136497 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4dbb97f8-c6bc-44e9-b468-e3577bb468af-ovnkube-script-lib\") pod \"ovnkube-node-nh6qf\" (UID: \"4dbb97f8-c6bc-44e9-b468-e3577bb468af\") " pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" Jul 27 17:28:04.137621 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.136542 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9d10075-b019-4d6d-9904-bdccbc7e8ca1-host-var-lib-kubelet\") pod \"multus-4mbmr\" (UID: \"b9d10075-b019-4d6d-9904-bdccbc7e8ca1\") " pod="openshift-multus/multus-4mbmr" Jul 27 17:28:04.137621 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.136545 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b9d10075-b019-4d6d-9904-bdccbc7e8ca1-host-run-multus-certs\") pod \"multus-4mbmr\" (UID: \"b9d10075-b019-4d6d-9904-bdccbc7e8ca1\") " pod="openshift-multus/multus-4mbmr" Jul 27 17:28:04.137621 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.136569 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfq85\" (UniqueName: \"kubernetes.io/projected/5115219e-8201-45d1-8a3f-6db85d5e741d-kube-api-access-lfq85\") pod \"network-metrics-daemon-69zjf\" (UID: \"5115219e-8201-45d1-8a3f-6db85d5e741d\") " pod="openshift-multus/network-metrics-daemon-69zjf" Jul 27 17:28:04.137621 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.136591 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b9d10075-b019-4d6d-9904-bdccbc7e8ca1-hostroot\") pod \"multus-4mbmr\" (UID: \"b9d10075-b019-4d6d-9904-bdccbc7e8ca1\") " pod="openshift-multus/multus-4mbmr" Jul 27 17:28:04.137621 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.136594 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c1bf323d-120b-45ff-9117-f8142039ccfc-cni-binary-copy\") pod \"multus-additional-cni-plugins-qp76t\" (UID: \"c1bf323d-120b-45ff-9117-f8142039ccfc\") " pod="openshift-multus/multus-additional-cni-plugins-qp76t" Jul 27 17:28:04.137621 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.136630 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b9d10075-b019-4d6d-9904-bdccbc7e8ca1-host-run-netns\") pod \"multus-4mbmr\" (UID: \"b9d10075-b019-4d6d-9904-bdccbc7e8ca1\") " pod="openshift-multus/multus-4mbmr" Jul 27 17:28:04.138372 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.136652 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c1bf323d-120b-45ff-9117-f8142039ccfc-os-release\") pod \"multus-additional-cni-plugins-qp76t\" (UID: \"c1bf323d-120b-45ff-9117-f8142039ccfc\") " pod="openshift-multus/multus-additional-cni-plugins-qp76t" Jul 27 17:28:04.138372 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.136659 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b9d10075-b019-4d6d-9904-bdccbc7e8ca1-multus-conf-dir\") pod \"multus-4mbmr\" (UID: \"b9d10075-b019-4d6d-9904-bdccbc7e8ca1\") " pod="openshift-multus/multus-4mbmr" Jul 27 17:28:04.138372 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.136699 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9d10075-b019-4d6d-9904-bdccbc7e8ca1-host-var-lib-kubelet\") pod \"multus-4mbmr\" (UID: \"b9d10075-b019-4d6d-9904-bdccbc7e8ca1\") " pod="openshift-multus/multus-4mbmr" Jul 27 17:28:04.138372 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.136699 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4dbb97f8-c6bc-44e9-b468-e3577bb468af-env-overrides\") pod \"ovnkube-node-nh6qf\" (UID: \"4dbb97f8-c6bc-44e9-b468-e3577bb468af\") " pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" Jul 27 17:28:04.138372 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.136737 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b9d10075-b019-4d6d-9904-bdccbc7e8ca1-multus-conf-dir\") pod \"multus-4mbmr\" (UID: \"b9d10075-b019-4d6d-9904-bdccbc7e8ca1\") " pod="openshift-multus/multus-4mbmr" Jul 27 17:28:04.138372 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.136744 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7qfl\" (UniqueName: \"kubernetes.io/projected/acba0339-7147-4770-8e0b-ae7fff7eba7e-kube-api-access-k7qfl\") pod \"node-ca-f5x5s\" (UID: \"acba0339-7147-4770-8e0b-ae7fff7eba7e\") " pod="openshift-image-registry/node-ca-f5x5s" Jul 27 17:28:04.138372 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.136703 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b9d10075-b019-4d6d-9904-bdccbc7e8ca1-host-run-netns\") pod \"multus-4mbmr\" (UID: \"b9d10075-b019-4d6d-9904-bdccbc7e8ca1\") " pod="openshift-multus/multus-4mbmr" Jul 27 17:28:04.138372 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.136801 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b9d10075-b019-4d6d-9904-bdccbc7e8ca1-cni-binary-copy\") pod \"multus-4mbmr\" (UID: \"b9d10075-b019-4d6d-9904-bdccbc7e8ca1\") " pod="openshift-multus/multus-4mbmr" Jul 27 17:28:04.138372 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.136812 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c1bf323d-120b-45ff-9117-f8142039ccfc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qp76t\" (UID: \"c1bf323d-120b-45ff-9117-f8142039ccfc\") " pod="openshift-multus/multus-additional-cni-plugins-qp76t" Jul 27 17:28:04.138372 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.136845 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c1bf323d-120b-45ff-9117-f8142039ccfc-cnibin\") pod \"multus-additional-cni-plugins-qp76t\" (UID: \"c1bf323d-120b-45ff-9117-f8142039ccfc\") " pod="openshift-multus/multus-additional-cni-plugins-qp76t" Jul 27 17:28:04.138372 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.136866 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b9d10075-b019-4d6d-9904-bdccbc7e8ca1-multus-daemon-config\") pod \"multus-4mbmr\" (UID: \"b9d10075-b019-4d6d-9904-bdccbc7e8ca1\") " pod="openshift-multus/multus-4mbmr" Jul 27 17:28:04.138372 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.136881 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swcl5\" (UniqueName: \"kubernetes.io/projected/c1bf323d-120b-45ff-9117-f8142039ccfc-kube-api-access-swcl5\") pod \"multus-additional-cni-plugins-qp76t\" (UID: \"c1bf323d-120b-45ff-9117-f8142039ccfc\") " pod="openshift-multus/multus-additional-cni-plugins-qp76t" Jul 27 17:28:04.138372 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.136932 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9d10075-b019-4d6d-9904-bdccbc7e8ca1-etc-kubernetes\") pod \"multus-4mbmr\" (UID: \"b9d10075-b019-4d6d-9904-bdccbc7e8ca1\") " pod="openshift-multus/multus-4mbmr" Jul 27 17:28:04.138372 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.136966 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9d10075-b019-4d6d-9904-bdccbc7e8ca1-etc-kubernetes\") pod \"multus-4mbmr\" (UID: \"b9d10075-b019-4d6d-9904-bdccbc7e8ca1\") " pod="openshift-multus/multus-4mbmr" Jul 27 17:28:04.138372 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.136998 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0e66277b-cc74-421a-84d7-d1d43ce8de57-konnectivity-ca\") pod \"konnectivity-agent-l5bjn\" (UID: \"0e66277b-cc74-421a-84d7-d1d43ce8de57\") " pod="kube-system/konnectivity-agent-l5bjn" Jul 27 17:28:04.138372 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.137028 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4dbb97f8-c6bc-44e9-b468-e3577bb468af-host-run-ovn-kubernetes\") pod \"ovnkube-node-nh6qf\" (UID: \"4dbb97f8-c6bc-44e9-b468-e3577bb468af\") " pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" Jul 27 17:28:04.138372 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.137113 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/acba0339-7147-4770-8e0b-ae7fff7eba7e-serviceca\") pod \"node-ca-f5x5s\" (UID: \"acba0339-7147-4770-8e0b-ae7fff7eba7e\") " pod="openshift-image-registry/node-ca-f5x5s" Jul 27 17:28:04.138372 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.137141 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4dbb97f8-c6bc-44e9-b468-e3577bb468af-systemd-units\") pod \"ovnkube-node-nh6qf\" (UID: \"4dbb97f8-c6bc-44e9-b468-e3577bb468af\") " pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" Jul 27 17:28:04.138957 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.137144 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4dbb97f8-c6bc-44e9-b468-e3577bb468af-host-run-ovn-kubernetes\") pod \"ovnkube-node-nh6qf\" (UID: \"4dbb97f8-c6bc-44e9-b468-e3577bb468af\") " pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" Jul 27 17:28:04.138957 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.137140 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8d2f0e3-917e-4be9-adf5-e233e4f97048-tmp-dir\") pod \"node-resolver-f22lz\" (UID: \"b8d2f0e3-917e-4be9-adf5-e233e4f97048\") " pod="openshift-dns/node-resolver-f22lz" Jul 27 17:28:04.138957 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.137192 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4dbb97f8-c6bc-44e9-b468-e3577bb468af-systemd-units\") pod \"ovnkube-node-nh6qf\" (UID: \"4dbb97f8-c6bc-44e9-b468-e3577bb468af\") " pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" Jul 27 17:28:04.138957 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.137252 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4dbb97f8-c6bc-44e9-b468-e3577bb468af-ovnkube-script-lib\") pod \"ovnkube-node-nh6qf\" (UID: \"4dbb97f8-c6bc-44e9-b468-e3577bb468af\") " pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" Jul 27 17:28:04.138957 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.137291 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c1bf323d-120b-45ff-9117-f8142039ccfc-cni-binary-copy\") pod \"multus-additional-cni-plugins-qp76t\" (UID: \"c1bf323d-120b-45ff-9117-f8142039ccfc\") " pod="openshift-multus/multus-additional-cni-plugins-qp76t" Jul 27 17:28:04.138957 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.137403 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q5ss\" (UniqueName: \"kubernetes.io/projected/b8d2f0e3-917e-4be9-adf5-e233e4f97048-kube-api-access-5q5ss\") pod \"node-resolver-f22lz\" (UID: \"b8d2f0e3-917e-4be9-adf5-e233e4f97048\") " pod="openshift-dns/node-resolver-f22lz" Jul 27 17:28:04.138957 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.137409 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4dbb97f8-c6bc-44e9-b468-e3577bb468af-ovnkube-config\") pod \"ovnkube-node-nh6qf\" (UID: \"4dbb97f8-c6bc-44e9-b468-e3577bb468af\") " pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" Jul 27 17:28:04.138957 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.137435 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5115219e-8201-45d1-8a3f-6db85d5e741d-metrics-certs\") pod \"network-metrics-daemon-69zjf\" (UID: \"5115219e-8201-45d1-8a3f-6db85d5e741d\") " pod="openshift-multus/network-metrics-daemon-69zjf" Jul 27 17:28:04.138957 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.137461 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c1bf323d-120b-45ff-9117-f8142039ccfc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qp76t\" (UID: \"c1bf323d-120b-45ff-9117-f8142039ccfc\") " pod="openshift-multus/multus-additional-cni-plugins-qp76t" Jul 27 17:28:04.138957 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.137488 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4dbb97f8-c6bc-44e9-b468-e3577bb468af-var-lib-openvswitch\") pod \"ovnkube-node-nh6qf\" (UID: \"4dbb97f8-c6bc-44e9-b468-e3577bb468af\") " pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" Jul 27 17:28:04.138957 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:04.137576 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 17:28:04.138957 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.137612 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/acba0339-7147-4770-8e0b-ae7fff7eba7e-serviceca\") pod \"node-ca-f5x5s\" (UID: \"acba0339-7147-4770-8e0b-ae7fff7eba7e\") " pod="openshift-image-registry/node-ca-f5x5s" Jul 27 17:28:04.138957 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:04.137648 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5115219e-8201-45d1-8a3f-6db85d5e741d-metrics-certs podName:5115219e-8201-45d1-8a3f-6db85d5e741d nodeName:}" failed. No retries permitted until 2026-07-27 17:28:04.637630503 +0000 UTC m=+2.078845457 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5115219e-8201-45d1-8a3f-6db85d5e741d-metrics-certs") pod "network-metrics-daemon-69zjf" (UID: "5115219e-8201-45d1-8a3f-6db85d5e741d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 17:28:04.138957 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.137677 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4dbb97f8-c6bc-44e9-b468-e3577bb468af-var-lib-openvswitch\") pod \"ovnkube-node-nh6qf\" (UID: \"4dbb97f8-c6bc-44e9-b468-e3577bb468af\") " pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" Jul 27 17:28:04.138957 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.137699 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0e66277b-cc74-421a-84d7-d1d43ce8de57-konnectivity-ca\") pod \"konnectivity-agent-l5bjn\" (UID: \"0e66277b-cc74-421a-84d7-d1d43ce8de57\") " pod="kube-system/konnectivity-agent-l5bjn" Jul 27 17:28:04.138957 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.137675 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c1bf323d-120b-45ff-9117-f8142039ccfc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qp76t\" (UID: \"c1bf323d-120b-45ff-9117-f8142039ccfc\") " pod="openshift-multus/multus-additional-cni-plugins-qp76t" Jul 27 17:28:04.138957 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.137737 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9b6c93b0-6e89-41e6-accd-cbacae11cf4e-iptables-alerter-script\") pod \"iptables-alerter-8smpx\" (UID: \"9b6c93b0-6e89-41e6-accd-cbacae11cf4e\") " pod="openshift-network-operator/iptables-alerter-8smpx" Jul 27 17:28:04.139455 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.137760 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6hl8\" (UniqueName: \"kubernetes.io/projected/9b6c93b0-6e89-41e6-accd-cbacae11cf4e-kube-api-access-n6hl8\") pod \"iptables-alerter-8smpx\" (UID: \"9b6c93b0-6e89-41e6-accd-cbacae11cf4e\") " pod="openshift-network-operator/iptables-alerter-8smpx" Jul 27 17:28:04.139455 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.137880 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b9d10075-b019-4d6d-9904-bdccbc7e8ca1-host-run-k8s-cni-cncf-io\") pod \"multus-4mbmr\" (UID: \"b9d10075-b019-4d6d-9904-bdccbc7e8ca1\") " pod="openshift-multus/multus-4mbmr" Jul 27 17:28:04.139455 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.137894 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c1bf323d-120b-45ff-9117-f8142039ccfc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qp76t\" (UID: \"c1bf323d-120b-45ff-9117-f8142039ccfc\") " pod="openshift-multus/multus-additional-cni-plugins-qp76t" Jul 27 17:28:04.139455 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.137913 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4dbb97f8-c6bc-44e9-b468-e3577bb468af-host-kubelet\") pod \"ovnkube-node-nh6qf\" (UID: \"4dbb97f8-c6bc-44e9-b468-e3577bb468af\") " pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" Jul 27 17:28:04.139455 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.137932 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c1bf323d-120b-45ff-9117-f8142039ccfc-system-cni-dir\") pod \"multus-additional-cni-plugins-qp76t\" (UID: \"c1bf323d-120b-45ff-9117-f8142039ccfc\") " pod="openshift-multus/multus-additional-cni-plugins-qp76t" Jul 27 17:28:04.139455 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.137943 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b9d10075-b019-4d6d-9904-bdccbc7e8ca1-host-run-k8s-cni-cncf-io\") pod \"multus-4mbmr\" (UID: \"b9d10075-b019-4d6d-9904-bdccbc7e8ca1\") " pod="openshift-multus/multus-4mbmr" Jul 27 17:28:04.139455 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.137955 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b9d10075-b019-4d6d-9904-bdccbc7e8ca1-host-var-lib-cni-multus\") pod \"multus-4mbmr\" (UID: \"b9d10075-b019-4d6d-9904-bdccbc7e8ca1\") " pod="openshift-multus/multus-4mbmr" Jul 27 17:28:04.139455 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.137983 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4dbb97f8-c6bc-44e9-b468-e3577bb468af-host-kubelet\") pod \"ovnkube-node-nh6qf\" (UID: \"4dbb97f8-c6bc-44e9-b468-e3577bb468af\") " pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" Jul 27 17:28:04.139455 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.137985 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0e66277b-cc74-421a-84d7-d1d43ce8de57-agent-certs\") pod \"konnectivity-agent-l5bjn\" (UID: \"0e66277b-cc74-421a-84d7-d1d43ce8de57\") " pod="kube-system/konnectivity-agent-l5bjn" Jul 27 17:28:04.139455 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.137999 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c1bf323d-120b-45ff-9117-f8142039ccfc-system-cni-dir\") pod \"multus-additional-cni-plugins-qp76t\" (UID: \"c1bf323d-120b-45ff-9117-f8142039ccfc\") " pod="openshift-multus/multus-additional-cni-plugins-qp76t" Jul 27 17:28:04.139455 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.138018 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p4xw\" (UniqueName: \"kubernetes.io/projected/ec0c6ea9-94a4-41b6-9e1f-cccf978294ab-kube-api-access-5p4xw\") pod \"network-check-target-jhzgt\" (UID: \"ec0c6ea9-94a4-41b6-9e1f-cccf978294ab\") " pod="openshift-network-diagnostics/network-check-target-jhzgt" Jul 27 17:28:04.139455 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.138036 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c1bf323d-120b-45ff-9117-f8142039ccfc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qp76t\" (UID: \"c1bf323d-120b-45ff-9117-f8142039ccfc\") " pod="openshift-multus/multus-additional-cni-plugins-qp76t" Jul 27 17:28:04.139455 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.138044 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4dbb97f8-c6bc-44e9-b468-e3577bb468af-host-slash\") pod \"ovnkube-node-nh6qf\" (UID: \"4dbb97f8-c6bc-44e9-b468-e3577bb468af\") " pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" Jul 27 17:28:04.139455 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.138017 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b9d10075-b019-4d6d-9904-bdccbc7e8ca1-host-var-lib-cni-multus\") pod \"multus-4mbmr\" (UID: \"b9d10075-b019-4d6d-9904-bdccbc7e8ca1\") " pod="openshift-multus/multus-4mbmr" Jul 27 17:28:04.139455 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.138070 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4dbb97f8-c6bc-44e9-b468-e3577bb468af-log-socket\") pod \"ovnkube-node-nh6qf\" (UID: \"4dbb97f8-c6bc-44e9-b468-e3577bb468af\") " pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" Jul 27 17:28:04.139455 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.138101 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b9d10075-b019-4d6d-9904-bdccbc7e8ca1-host-var-lib-cni-bin\") pod \"multus-4mbmr\" (UID: \"b9d10075-b019-4d6d-9904-bdccbc7e8ca1\") " pod="openshift-multus/multus-4mbmr" Jul 27 17:28:04.139455 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.138125 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4dbb97f8-c6bc-44e9-b468-e3577bb468af-etc-openvswitch\") pod \"ovnkube-node-nh6qf\" (UID: \"4dbb97f8-c6bc-44e9-b468-e3577bb468af\") " pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" Jul 27 17:28:04.139979 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.138130 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b9d10075-b019-4d6d-9904-bdccbc7e8ca1-host-var-lib-cni-bin\") pod \"multus-4mbmr\" (UID: \"b9d10075-b019-4d6d-9904-bdccbc7e8ca1\") " pod="openshift-multus/multus-4mbmr" Jul 27 17:28:04.139979 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.138125 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4dbb97f8-c6bc-44e9-b468-e3577bb468af-log-socket\") pod \"ovnkube-node-nh6qf\" (UID: \"4dbb97f8-c6bc-44e9-b468-e3577bb468af\") " pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" Jul 27 17:28:04.139979 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.138159 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4dbb97f8-c6bc-44e9-b468-e3577bb468af-etc-openvswitch\") pod \"ovnkube-node-nh6qf\" (UID: \"4dbb97f8-c6bc-44e9-b468-e3577bb468af\") " pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" Jul 27 17:28:04.139979 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.138168 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4dbb97f8-c6bc-44e9-b468-e3577bb468af-ovn-node-metrics-cert\") pod \"ovnkube-node-nh6qf\" (UID: \"4dbb97f8-c6bc-44e9-b468-e3577bb468af\") " pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" Jul 27 17:28:04.139979 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.138249 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4dbb97f8-c6bc-44e9-b468-e3577bb468af-host-slash\") pod \"ovnkube-node-nh6qf\" (UID: \"4dbb97f8-c6bc-44e9-b468-e3577bb468af\") " pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" Jul 27 17:28:04.139979 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.138269 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9b6c93b0-6e89-41e6-accd-cbacae11cf4e-iptables-alerter-script\") pod \"iptables-alerter-8smpx\" (UID: \"9b6c93b0-6e89-41e6-accd-cbacae11cf4e\") " pod="openshift-network-operator/iptables-alerter-8smpx" Jul 27 17:28:04.140279 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.140150 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4dbb97f8-c6bc-44e9-b468-e3577bb468af-ovn-node-metrics-cert\") pod \"ovnkube-node-nh6qf\" (UID: \"4dbb97f8-c6bc-44e9-b468-e3577bb468af\") " pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" Jul 27 17:28:04.140279 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.140264 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0e66277b-cc74-421a-84d7-d1d43ce8de57-agent-certs\") pod \"konnectivity-agent-l5bjn\" (UID: \"0e66277b-cc74-421a-84d7-d1d43ce8de57\") " pod="kube-system/konnectivity-agent-l5bjn" Jul 27 17:28:04.146716 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:04.146692 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 17:28:04.146716 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:04.146714 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 17:28:04.146944 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:04.146725 2570 projected.go:196] Error preparing data for projected volume kube-api-access-5p4xw for pod openshift-network-diagnostics/network-check-target-jhzgt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 17:28:04.146944 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:04.146778 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec0c6ea9-94a4-41b6-9e1f-cccf978294ab-kube-api-access-5p4xw podName:ec0c6ea9-94a4-41b6-9e1f-cccf978294ab nodeName:}" failed. No retries permitted until 2026-07-27 17:28:04.646757989 +0000 UTC m=+2.087972944 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-5p4xw" (UniqueName: "kubernetes.io/projected/ec0c6ea9-94a4-41b6-9e1f-cccf978294ab-kube-api-access-5p4xw") pod "network-check-target-jhzgt" (UID: "ec0c6ea9-94a4-41b6-9e1f-cccf978294ab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 17:28:04.148838 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.148809 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q5ss\" (UniqueName: \"kubernetes.io/projected/b8d2f0e3-917e-4be9-adf5-e233e4f97048-kube-api-access-5q5ss\") pod \"node-resolver-f22lz\" (UID: \"b8d2f0e3-917e-4be9-adf5-e233e4f97048\") " pod="openshift-dns/node-resolver-f22lz" Jul 27 17:28:04.148985 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.148841 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7spf\" (UniqueName: \"kubernetes.io/projected/4dbb97f8-c6bc-44e9-b468-e3577bb468af-kube-api-access-q7spf\") pod \"ovnkube-node-nh6qf\" (UID: \"4dbb97f8-c6bc-44e9-b468-e3577bb468af\") " pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" Jul 27 17:28:04.149192 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.149170 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6hl8\" (UniqueName: \"kubernetes.io/projected/9b6c93b0-6e89-41e6-accd-cbacae11cf4e-kube-api-access-n6hl8\") pod \"iptables-alerter-8smpx\" (UID: \"9b6c93b0-6e89-41e6-accd-cbacae11cf4e\") " pod="openshift-network-operator/iptables-alerter-8smpx" Jul 27 17:28:04.149508 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.149489 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7qfl\" (UniqueName: \"kubernetes.io/projected/acba0339-7147-4770-8e0b-ae7fff7eba7e-kube-api-access-k7qfl\") pod \"node-ca-f5x5s\" (UID: \"acba0339-7147-4770-8e0b-ae7fff7eba7e\") " pod="openshift-image-registry/node-ca-f5x5s" Jul 27 17:28:04.149608 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.149503 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swcl5\" (UniqueName: \"kubernetes.io/projected/c1bf323d-120b-45ff-9117-f8142039ccfc-kube-api-access-swcl5\") pod \"multus-additional-cni-plugins-qp76t\" (UID: \"c1bf323d-120b-45ff-9117-f8142039ccfc\") " pod="openshift-multus/multus-additional-cni-plugins-qp76t" Jul 27 17:28:04.149608 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.149488 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txgkl\" (UniqueName: \"kubernetes.io/projected/b9d10075-b019-4d6d-9904-bdccbc7e8ca1-kube-api-access-txgkl\") pod \"multus-4mbmr\" (UID: \"b9d10075-b019-4d6d-9904-bdccbc7e8ca1\") " pod="openshift-multus/multus-4mbmr" Jul 27 17:28:04.149608 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.149556 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfq85\" (UniqueName: \"kubernetes.io/projected/5115219e-8201-45d1-8a3f-6db85d5e741d-kube-api-access-lfq85\") pod \"network-metrics-daemon-69zjf\" (UID: \"5115219e-8201-45d1-8a3f-6db85d5e741d\") " pod="openshift-multus/network-metrics-daemon-69zjf" Jul 27 17:28:04.248894 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.248822 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lnst" Jul 27 17:28:04.258721 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.258693 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-w4d45" Jul 27 17:28:04.278444 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.278421 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-f22lz" Jul 27 17:28:04.282851 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:04.282805 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96f53b3100180b7e57940a1d1a844aea.slice/crio-3939274eec257a9a498d891892ed5ca81ce1397768cd3cbb395b2ac7fd034597 WatchSource:0}: Error finding container 3939274eec257a9a498d891892ed5ca81ce1397768cd3cbb395b2ac7fd034597: Status 404 returned error can't find the container with id 3939274eec257a9a498d891892ed5ca81ce1397768cd3cbb395b2ac7fd034597 Jul 27 17:28:04.283079 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:04.283050 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44c736a9_a8db_4d86_8fb1_bb5287d8c319.slice/crio-b5de52c96e3fdba7f09ce02f9b9504d3b0da56421e1e1a04c81267d02ce6bd5d WatchSource:0}: Error finding container b5de52c96e3fdba7f09ce02f9b9504d3b0da56421e1e1a04c81267d02ce6bd5d: Status 404 returned error can't find the container with id b5de52c96e3fdba7f09ce02f9b9504d3b0da56421e1e1a04c81267d02ce6bd5d Jul 27 17:28:04.284205 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:04.284179 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fd9dbff_b2d0_4c8b_a65a_8f9434f632c9.slice/crio-a67b3765ff7d42f69ef4ac2050aa6037f2af506f0e6b7c055c6a72dcdc1e1a3f WatchSource:0}: Error finding container a67b3765ff7d42f69ef4ac2050aa6037f2af506f0e6b7c055c6a72dcdc1e1a3f: Status 404 returned error can't find the container with id a67b3765ff7d42f69ef4ac2050aa6037f2af506f0e6b7c055c6a72dcdc1e1a3f Jul 27 17:28:04.284635 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:04.284612 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92aa1923735291b61478f523669c7af8.slice/crio-9fdba4522ee8abde7444e860eea30223eed48708226e34c6210d2223165ed489 WatchSource:0}: Error finding container 9fdba4522ee8abde7444e860eea30223eed48708226e34c6210d2223165ed489: Status 404 returned error can't find the container with id 9fdba4522ee8abde7444e860eea30223eed48708226e34c6210d2223165ed489 Jul 27 17:28:04.288099 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.288081 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 27 17:28:04.289810 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:04.289794 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8d2f0e3_917e_4be9_adf5_e233e4f97048.slice/crio-20361122c430afbaf2dfc5f9805bf44bf579168cdb12a5fcd4df8f99ba73a500 WatchSource:0}: Error finding container 20361122c430afbaf2dfc5f9805bf44bf579168cdb12a5fcd4df8f99ba73a500: Status 404 returned error can't find the container with id 20361122c430afbaf2dfc5f9805bf44bf579168cdb12a5fcd4df8f99ba73a500 Jul 27 17:28:04.295318 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.295297 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-f5x5s" Jul 27 17:28:04.302383 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.302362 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4mbmr" Jul 27 17:28:04.305639 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:04.305612 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacba0339_7147_4770_8e0b_ae7fff7eba7e.slice/crio-0e14de571f0d349ea352f125d2a8fde53ae9b83a9a14784d15e200d473cd7522 WatchSource:0}: Error finding container 0e14de571f0d349ea352f125d2a8fde53ae9b83a9a14784d15e200d473cd7522: Status 404 returned error can't find the container with id 0e14de571f0d349ea352f125d2a8fde53ae9b83a9a14784d15e200d473cd7522 Jul 27 17:28:04.310900 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:04.310876 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9d10075_b019_4d6d_9904_bdccbc7e8ca1.slice/crio-a1d06ab6e0becc4ba812110ef69608e54f34a66f419e62d8e91a72eb747acd92 WatchSource:0}: Error finding container a1d06ab6e0becc4ba812110ef69608e54f34a66f419e62d8e91a72eb747acd92: Status 404 returned error can't find the container with id a1d06ab6e0becc4ba812110ef69608e54f34a66f419e62d8e91a72eb747acd92 Jul 27 17:28:04.325426 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.325397 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-l5bjn" Jul 27 17:28:04.334386 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:04.334363 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e66277b_cc74_421a_84d7_d1d43ce8de57.slice/crio-c03422a9e6cb21c4e0e6d06d11c7d5077a2b41cad1182b8c47e34e24f6ef6c00 WatchSource:0}: Error finding container c03422a9e6cb21c4e0e6d06d11c7d5077a2b41cad1182b8c47e34e24f6ef6c00: Status 404 returned error can't find the container with id c03422a9e6cb21c4e0e6d06d11c7d5077a2b41cad1182b8c47e34e24f6ef6c00 Jul 27 17:28:04.337305 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.337285 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qp76t" Jul 27 17:28:04.346894 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:04.346873 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1bf323d_120b_45ff_9117_f8142039ccfc.slice/crio-e118ed00be680bc4af4d36d1767f3da06c891ed85b44c990403378d935496ae4 WatchSource:0}: Error finding container e118ed00be680bc4af4d36d1767f3da06c891ed85b44c990403378d935496ae4: Status 404 returned error can't find the container with id e118ed00be680bc4af4d36d1767f3da06c891ed85b44c990403378d935496ae4 Jul 27 17:28:04.351744 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.351726 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8smpx" Jul 27 17:28:04.355178 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.355163 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" Jul 27 17:28:04.360022 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:04.359997 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b6c93b0_6e89_41e6_accd_cbacae11cf4e.slice/crio-52159083a8df5c59e0bf16794fa64ea7c6081efc06ab2f4df7135f8aff3d5ff9 WatchSource:0}: Error finding container 52159083a8df5c59e0bf16794fa64ea7c6081efc06ab2f4df7135f8aff3d5ff9: Status 404 returned error can't find the container with id 52159083a8df5c59e0bf16794fa64ea7c6081efc06ab2f4df7135f8aff3d5ff9 Jul 27 17:28:04.363937 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:04.363916 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dbb97f8_c6bc_44e9_b468_e3577bb468af.slice/crio-5919228742ca04f6bf57d8f2d193ea4223f8457e1c5cbf7df2dd4cee15595851 WatchSource:0}: Error finding container 5919228742ca04f6bf57d8f2d193ea4223f8457e1c5cbf7df2dd4cee15595851: Status 404 returned error can't find the container with id 5919228742ca04f6bf57d8f2d193ea4223f8457e1c5cbf7df2dd4cee15595851 Jul 27 17:28:04.365437 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:04.365419 2570 kuberuntime_manager.go:1459] "Unhandled Error" err="container ovn-controller start failed in pod ovnkube-node-nh6qf_openshift-ovn-kubernetes(4dbb97f8-c6bc-44e9-b468-e3577bb468af): ErrImagePull: pull QPS exceeded" logger="UnhandledError" Jul 27 17:28:04.366567 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:04.366550 2570 kuberuntime_manager.go:1459] "Unhandled Error" err="container kube-rbac-proxy-node start failed in pod ovnkube-node-nh6qf_openshift-ovn-kubernetes(4dbb97f8-c6bc-44e9-b468-e3577bb468af): ErrImagePull: pull QPS exceeded" logger="UnhandledError" Jul 27 17:28:04.369809 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:04.369772 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"ovn-controller\" with ErrImagePull: \"pull QPS exceeded\", failed to \"StartContainer\" for \"ovn-acl-logging\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c61b1bd630d745ebd63d45ede0bae739076b4213e12bea2e0bb124e8f4727ab0\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"kube-rbac-proxy-node\" with ErrImagePull: \"pull QPS exceeded\", failed to \"StartContainer\" for \"kube-rbac-proxy-ovn-metrics\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79363f7abd34060a04015eceb060626d31d91d8c0aa51ce25ec751c237169fdc\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"northd\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c61b1bd630d745ebd63d45ede0bae739076b4213e12bea2e0bb124e8f4727ab0\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"nbdb\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c61b1bd630d745ebd63d45ede0bae739076b4213e12bea2e0bb124e8f4727ab0\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"sbdb\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c61b1bd630d745ebd63d45ede0bae739076b4213e12bea2e0bb124e8f4727ab0\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"ovnkube-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c61b1bd630d745ebd63d45ede0bae739076b4213e12bea2e0bb124e8f4727ab0\\\": ErrImagePull: pull QPS exceeded\"]" pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" podUID="4dbb97f8-c6bc-44e9-b468-e3577bb468af" Jul 27 17:28:04.541548 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.541450 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7209ee7f-3019-4465-8175-3317741f9b09-original-pull-secret\") pod \"global-pull-secret-syncer-d4nqw\" (UID: \"7209ee7f-3019-4465-8175-3317741f9b09\") " pod="kube-system/global-pull-secret-syncer-d4nqw" Jul 27 17:28:04.541687 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:04.541607 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 17:28:04.541687 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:04.541669 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7209ee7f-3019-4465-8175-3317741f9b09-original-pull-secret podName:7209ee7f-3019-4465-8175-3317741f9b09 nodeName:}" failed. No retries permitted until 2026-07-27 17:28:05.541654437 +0000 UTC m=+2.982869390 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7209ee7f-3019-4465-8175-3317741f9b09-original-pull-secret") pod "global-pull-secret-syncer-d4nqw" (UID: "7209ee7f-3019-4465-8175-3317741f9b09") : object "kube-system"/"original-pull-secret" not registered Jul 27 17:28:04.642780 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.642745 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5115219e-8201-45d1-8a3f-6db85d5e741d-metrics-certs\") pod \"network-metrics-daemon-69zjf\" (UID: \"5115219e-8201-45d1-8a3f-6db85d5e741d\") " pod="openshift-multus/network-metrics-daemon-69zjf" Jul 27 17:28:04.642945 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:04.642859 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 17:28:04.642945 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:04.642910 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5115219e-8201-45d1-8a3f-6db85d5e741d-metrics-certs podName:5115219e-8201-45d1-8a3f-6db85d5e741d nodeName:}" failed. No retries permitted until 2026-07-27 17:28:05.642895717 +0000 UTC m=+3.084110669 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5115219e-8201-45d1-8a3f-6db85d5e741d-metrics-certs") pod "network-metrics-daemon-69zjf" (UID: "5115219e-8201-45d1-8a3f-6db85d5e741d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 17:28:04.743259 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.743225 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p4xw\" (UniqueName: \"kubernetes.io/projected/ec0c6ea9-94a4-41b6-9e1f-cccf978294ab-kube-api-access-5p4xw\") pod \"network-check-target-jhzgt\" (UID: \"ec0c6ea9-94a4-41b6-9e1f-cccf978294ab\") " pod="openshift-network-diagnostics/network-check-target-jhzgt" Jul 27 17:28:04.743438 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:04.743381 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 17:28:04.743438 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:04.743400 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 17:28:04.743438 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:04.743412 2570 projected.go:196] Error preparing data for projected volume kube-api-access-5p4xw for pod openshift-network-diagnostics/network-check-target-jhzgt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 17:28:04.743633 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:04.743475 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec0c6ea9-94a4-41b6-9e1f-cccf978294ab-kube-api-access-5p4xw podName:ec0c6ea9-94a4-41b6-9e1f-cccf978294ab nodeName:}" failed. No retries permitted until 2026-07-27 17:28:05.743455604 +0000 UTC m=+3.184670557 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-5p4xw" (UniqueName: "kubernetes.io/projected/ec0c6ea9-94a4-41b6-9e1f-cccf978294ab-kube-api-access-5p4xw") pod "network-check-target-jhzgt" (UID: "ec0c6ea9-94a4-41b6-9e1f-cccf978294ab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 17:28:04.970705 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.970595 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 17:23:03 +0000 UTC" deadline="2028-04-28 01:05:01.682013196 +0000 UTC" Jul 27 17:28:04.970705 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:04.970655 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15367h36m56.711361678s" Jul 27 17:28:05.124957 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:05.124314 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-69zjf" Jul 27 17:28:05.124957 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:05.124463 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-69zjf" podUID="5115219e-8201-45d1-8a3f-6db85d5e741d" Jul 27 17:28:05.132970 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:05.132880 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qp76t" event={"ID":"c1bf323d-120b-45ff-9117-f8142039ccfc","Type":"ContainerStarted","Data":"e118ed00be680bc4af4d36d1767f3da06c891ed85b44c990403378d935496ae4"} Jul 27 17:28:05.138566 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:05.138082 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4mbmr" event={"ID":"b9d10075-b019-4d6d-9904-bdccbc7e8ca1","Type":"ContainerStarted","Data":"a1d06ab6e0becc4ba812110ef69608e54f34a66f419e62d8e91a72eb747acd92"} Jul 27 17:28:05.141277 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:05.141244 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-f5x5s" event={"ID":"acba0339-7147-4770-8e0b-ae7fff7eba7e","Type":"ContainerStarted","Data":"0e14de571f0d349ea352f125d2a8fde53ae9b83a9a14784d15e200d473cd7522"} Jul 27 17:28:05.144716 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:05.144686 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-185.ec2.internal" event={"ID":"96f53b3100180b7e57940a1d1a844aea","Type":"ContainerStarted","Data":"3939274eec257a9a498d891892ed5ca81ce1397768cd3cbb395b2ac7fd034597"} Jul 27 17:28:05.147079 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:05.147051 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-l5bjn" event={"ID":"0e66277b-cc74-421a-84d7-d1d43ce8de57","Type":"ContainerStarted","Data":"c03422a9e6cb21c4e0e6d06d11c7d5077a2b41cad1182b8c47e34e24f6ef6c00"} Jul 27 17:28:05.149366 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:05.149339 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-f22lz" event={"ID":"b8d2f0e3-917e-4be9-adf5-e233e4f97048","Type":"ContainerStarted","Data":"20361122c430afbaf2dfc5f9805bf44bf579168cdb12a5fcd4df8f99ba73a500"} Jul 27 17:28:05.151244 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:05.151217 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-185.ec2.internal" event={"ID":"92aa1923735291b61478f523669c7af8","Type":"ContainerStarted","Data":"9fdba4522ee8abde7444e860eea30223eed48708226e34c6210d2223165ed489"} Jul 27 17:28:05.158151 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:05.158121 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-w4d45" event={"ID":"44c736a9-a8db-4d86-8fb1-bb5287d8c319","Type":"ContainerStarted","Data":"b5de52c96e3fdba7f09ce02f9b9504d3b0da56421e1e1a04c81267d02ce6bd5d"} Jul 27 17:28:05.161027 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:05.160991 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lnst" event={"ID":"3fd9dbff-b2d0-4c8b-a65a-8f9434f632c9","Type":"ContainerStarted","Data":"a67b3765ff7d42f69ef4ac2050aa6037f2af506f0e6b7c055c6a72dcdc1e1a3f"} Jul 27 17:28:05.164722 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:05.164658 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" event={"ID":"4dbb97f8-c6bc-44e9-b468-e3577bb468af","Type":"ContainerStarted","Data":"5919228742ca04f6bf57d8f2d193ea4223f8457e1c5cbf7df2dd4cee15595851"} Jul 27 17:28:05.172931 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:05.172904 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8smpx" event={"ID":"9b6c93b0-6e89-41e6-accd-cbacae11cf4e","Type":"ContainerStarted","Data":"52159083a8df5c59e0bf16794fa64ea7c6081efc06ab2f4df7135f8aff3d5ff9"} Jul 27 17:28:05.188415 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:05.188304 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"ovn-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c61b1bd630d745ebd63d45ede0bae739076b4213e12bea2e0bb124e8f4727ab0\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"ovn-acl-logging\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c61b1bd630d745ebd63d45ede0bae739076b4213e12bea2e0bb124e8f4727ab0\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"kube-rbac-proxy-node\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79363f7abd34060a04015eceb060626d31d91d8c0aa51ce25ec751c237169fdc\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"kube-rbac-proxy-ovn-metrics\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79363f7abd34060a04015eceb060626d31d91d8c0aa51ce25ec751c237169fdc\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"northd\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c61b1bd630d745ebd63d45ede0bae739076b4213e12bea2e0bb124e8f4727ab0\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"nbdb\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c61b1bd630d745ebd63d45ede0bae739076b4213e12bea2e0bb124e8f4727ab0\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"sbdb\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c61b1bd630d745ebd63d45ede0bae739076b4213e12bea2e0bb124e8f4727ab0\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"ovnkube-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c61b1bd630d745ebd63d45ede0bae739076b4213e12bea2e0bb124e8f4727ab0\\\": ErrImagePull: pull QPS exceeded\"]" pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" podUID="4dbb97f8-c6bc-44e9-b468-e3577bb468af" Jul 27 17:28:05.310790 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:05.310712 2570 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 17:28:05.548086 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:05.548038 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7209ee7f-3019-4465-8175-3317741f9b09-original-pull-secret\") pod \"global-pull-secret-syncer-d4nqw\" (UID: \"7209ee7f-3019-4465-8175-3317741f9b09\") " pod="kube-system/global-pull-secret-syncer-d4nqw" Jul 27 17:28:05.548284 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:05.548178 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 17:28:05.548284 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:05.548235 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7209ee7f-3019-4465-8175-3317741f9b09-original-pull-secret podName:7209ee7f-3019-4465-8175-3317741f9b09 nodeName:}" failed. No retries permitted until 2026-07-27 17:28:07.548218616 +0000 UTC m=+4.989433570 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7209ee7f-3019-4465-8175-3317741f9b09-original-pull-secret") pod "global-pull-secret-syncer-d4nqw" (UID: "7209ee7f-3019-4465-8175-3317741f9b09") : object "kube-system"/"original-pull-secret" not registered Jul 27 17:28:05.638570 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:05.638476 2570 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 17:28:05.648581 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:05.648551 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5115219e-8201-45d1-8a3f-6db85d5e741d-metrics-certs\") pod \"network-metrics-daemon-69zjf\" (UID: \"5115219e-8201-45d1-8a3f-6db85d5e741d\") " pod="openshift-multus/network-metrics-daemon-69zjf" Jul 27 17:28:05.648753 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:05.648726 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 17:28:05.648813 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:05.648788 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5115219e-8201-45d1-8a3f-6db85d5e741d-metrics-certs podName:5115219e-8201-45d1-8a3f-6db85d5e741d nodeName:}" failed. No retries permitted until 2026-07-27 17:28:07.648769474 +0000 UTC m=+5.089984431 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5115219e-8201-45d1-8a3f-6db85d5e741d-metrics-certs") pod "network-metrics-daemon-69zjf" (UID: "5115219e-8201-45d1-8a3f-6db85d5e741d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 17:28:05.749962 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:05.749930 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p4xw\" (UniqueName: \"kubernetes.io/projected/ec0c6ea9-94a4-41b6-9e1f-cccf978294ab-kube-api-access-5p4xw\") pod \"network-check-target-jhzgt\" (UID: \"ec0c6ea9-94a4-41b6-9e1f-cccf978294ab\") " pod="openshift-network-diagnostics/network-check-target-jhzgt" Jul 27 17:28:05.750165 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:05.750148 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 17:28:05.750225 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:05.750174 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 17:28:05.750225 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:05.750187 2570 projected.go:196] Error preparing data for projected volume kube-api-access-5p4xw for pod openshift-network-diagnostics/network-check-target-jhzgt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 17:28:05.750321 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:05.750245 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec0c6ea9-94a4-41b6-9e1f-cccf978294ab-kube-api-access-5p4xw podName:ec0c6ea9-94a4-41b6-9e1f-cccf978294ab nodeName:}" failed. No retries permitted until 2026-07-27 17:28:07.750227887 +0000 UTC m=+5.191442859 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-5p4xw" (UniqueName: "kubernetes.io/projected/ec0c6ea9-94a4-41b6-9e1f-cccf978294ab-kube-api-access-5p4xw") pod "network-check-target-jhzgt" (UID: "ec0c6ea9-94a4-41b6-9e1f-cccf978294ab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 17:28:05.971496 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:05.971404 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 17:23:03 +0000 UTC" deadline="2028-04-23 17:46:21.29182324 +0000 UTC" Jul 27 17:28:05.971496 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:05.971444 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15264h18m15.320383675s" Jul 27 17:28:06.123782 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:06.123750 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d4nqw" Jul 27 17:28:06.123970 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:06.123884 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d4nqw" podUID="7209ee7f-3019-4465-8175-3317741f9b09" Jul 27 17:28:06.124313 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:06.124292 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhzgt" Jul 27 17:28:06.124399 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:06.124386 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhzgt" podUID="ec0c6ea9-94a4-41b6-9e1f-cccf978294ab" Jul 27 17:28:06.582354 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:06.582322 2570 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 17:28:07.126289 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:07.126255 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-69zjf" Jul 27 17:28:07.126777 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:07.126405 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-69zjf" podUID="5115219e-8201-45d1-8a3f-6db85d5e741d" Jul 27 17:28:07.568855 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:07.568818 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7209ee7f-3019-4465-8175-3317741f9b09-original-pull-secret\") pod \"global-pull-secret-syncer-d4nqw\" (UID: \"7209ee7f-3019-4465-8175-3317741f9b09\") " pod="kube-system/global-pull-secret-syncer-d4nqw" Jul 27 17:28:07.569035 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:07.569019 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 17:28:07.569099 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:07.569085 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7209ee7f-3019-4465-8175-3317741f9b09-original-pull-secret podName:7209ee7f-3019-4465-8175-3317741f9b09 nodeName:}" failed. No retries permitted until 2026-07-27 17:28:11.569066155 +0000 UTC m=+9.010281108 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7209ee7f-3019-4465-8175-3317741f9b09-original-pull-secret") pod "global-pull-secret-syncer-d4nqw" (UID: "7209ee7f-3019-4465-8175-3317741f9b09") : object "kube-system"/"original-pull-secret" not registered Jul 27 17:28:07.669648 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:07.669600 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5115219e-8201-45d1-8a3f-6db85d5e741d-metrics-certs\") pod \"network-metrics-daemon-69zjf\" (UID: \"5115219e-8201-45d1-8a3f-6db85d5e741d\") " pod="openshift-multus/network-metrics-daemon-69zjf" Jul 27 17:28:07.669824 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:07.669801 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 17:28:07.669889 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:07.669876 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5115219e-8201-45d1-8a3f-6db85d5e741d-metrics-certs podName:5115219e-8201-45d1-8a3f-6db85d5e741d nodeName:}" failed. No retries permitted until 2026-07-27 17:28:11.669857244 +0000 UTC m=+9.111072200 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5115219e-8201-45d1-8a3f-6db85d5e741d-metrics-certs") pod "network-metrics-daemon-69zjf" (UID: "5115219e-8201-45d1-8a3f-6db85d5e741d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 17:28:07.771036 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:07.770996 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p4xw\" (UniqueName: \"kubernetes.io/projected/ec0c6ea9-94a4-41b6-9e1f-cccf978294ab-kube-api-access-5p4xw\") pod \"network-check-target-jhzgt\" (UID: \"ec0c6ea9-94a4-41b6-9e1f-cccf978294ab\") " pod="openshift-network-diagnostics/network-check-target-jhzgt" Jul 27 17:28:07.771202 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:07.771187 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 17:28:07.771263 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:07.771208 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 17:28:07.771263 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:07.771221 2570 projected.go:196] Error preparing data for projected volume kube-api-access-5p4xw for pod openshift-network-diagnostics/network-check-target-jhzgt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 17:28:07.771337 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:07.771288 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec0c6ea9-94a4-41b6-9e1f-cccf978294ab-kube-api-access-5p4xw podName:ec0c6ea9-94a4-41b6-9e1f-cccf978294ab nodeName:}" failed. No retries permitted until 2026-07-27 17:28:11.771271112 +0000 UTC m=+9.212486070 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-5p4xw" (UniqueName: "kubernetes.io/projected/ec0c6ea9-94a4-41b6-9e1f-cccf978294ab-kube-api-access-5p4xw") pod "network-check-target-jhzgt" (UID: "ec0c6ea9-94a4-41b6-9e1f-cccf978294ab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 17:28:08.123580 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:08.123547 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d4nqw" Jul 27 17:28:08.123580 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:08.123573 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhzgt" Jul 27 17:28:08.123780 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:08.123690 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d4nqw" podUID="7209ee7f-3019-4465-8175-3317741f9b09" Jul 27 17:28:08.123833 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:08.123814 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhzgt" podUID="ec0c6ea9-94a4-41b6-9e1f-cccf978294ab" Jul 27 17:28:09.127619 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:09.127584 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-69zjf" Jul 27 17:28:09.128074 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:09.127724 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-69zjf" podUID="5115219e-8201-45d1-8a3f-6db85d5e741d" Jul 27 17:28:10.123604 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:10.123567 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d4nqw" Jul 27 17:28:10.123784 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:10.123574 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhzgt" Jul 27 17:28:10.123784 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:10.123704 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d4nqw" podUID="7209ee7f-3019-4465-8175-3317741f9b09" Jul 27 17:28:10.123911 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:10.123779 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhzgt" podUID="ec0c6ea9-94a4-41b6-9e1f-cccf978294ab" Jul 27 17:28:11.124499 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:11.124465 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-69zjf" Jul 27 17:28:11.124968 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:11.124638 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-69zjf" podUID="5115219e-8201-45d1-8a3f-6db85d5e741d" Jul 27 17:28:11.602057 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:11.602027 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7209ee7f-3019-4465-8175-3317741f9b09-original-pull-secret\") pod \"global-pull-secret-syncer-d4nqw\" (UID: \"7209ee7f-3019-4465-8175-3317741f9b09\") " pod="kube-system/global-pull-secret-syncer-d4nqw" Jul 27 17:28:11.602278 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:11.602196 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 17:28:11.602278 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:11.602276 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7209ee7f-3019-4465-8175-3317741f9b09-original-pull-secret podName:7209ee7f-3019-4465-8175-3317741f9b09 nodeName:}" failed. No retries permitted until 2026-07-27 17:28:19.602254992 +0000 UTC m=+17.043469948 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7209ee7f-3019-4465-8175-3317741f9b09-original-pull-secret") pod "global-pull-secret-syncer-d4nqw" (UID: "7209ee7f-3019-4465-8175-3317741f9b09") : object "kube-system"/"original-pull-secret" not registered Jul 27 17:28:11.702597 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:11.702556 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5115219e-8201-45d1-8a3f-6db85d5e741d-metrics-certs\") pod \"network-metrics-daemon-69zjf\" (UID: \"5115219e-8201-45d1-8a3f-6db85d5e741d\") " pod="openshift-multus/network-metrics-daemon-69zjf" Jul 27 17:28:11.702769 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:11.702705 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 17:28:11.702833 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:11.702784 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5115219e-8201-45d1-8a3f-6db85d5e741d-metrics-certs podName:5115219e-8201-45d1-8a3f-6db85d5e741d nodeName:}" failed. No retries permitted until 2026-07-27 17:28:19.702765218 +0000 UTC m=+17.143980170 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5115219e-8201-45d1-8a3f-6db85d5e741d-metrics-certs") pod "network-metrics-daemon-69zjf" (UID: "5115219e-8201-45d1-8a3f-6db85d5e741d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 17:28:11.803846 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:11.803807 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p4xw\" (UniqueName: \"kubernetes.io/projected/ec0c6ea9-94a4-41b6-9e1f-cccf978294ab-kube-api-access-5p4xw\") pod \"network-check-target-jhzgt\" (UID: \"ec0c6ea9-94a4-41b6-9e1f-cccf978294ab\") " pod="openshift-network-diagnostics/network-check-target-jhzgt" Jul 27 17:28:11.804027 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:11.803930 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 17:28:11.804027 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:11.803946 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 17:28:11.804027 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:11.803956 2570 projected.go:196] Error preparing data for projected volume kube-api-access-5p4xw for pod openshift-network-diagnostics/network-check-target-jhzgt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 17:28:11.804027 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:11.804010 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec0c6ea9-94a4-41b6-9e1f-cccf978294ab-kube-api-access-5p4xw podName:ec0c6ea9-94a4-41b6-9e1f-cccf978294ab nodeName:}" failed. No retries permitted until 2026-07-27 17:28:19.803991508 +0000 UTC m=+17.245206466 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-5p4xw" (UniqueName: "kubernetes.io/projected/ec0c6ea9-94a4-41b6-9e1f-cccf978294ab-kube-api-access-5p4xw") pod "network-check-target-jhzgt" (UID: "ec0c6ea9-94a4-41b6-9e1f-cccf978294ab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 17:28:12.123859 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:12.123824 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhzgt" Jul 27 17:28:12.124030 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:12.123824 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d4nqw" Jul 27 17:28:12.124030 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:12.123954 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhzgt" podUID="ec0c6ea9-94a4-41b6-9e1f-cccf978294ab" Jul 27 17:28:12.124030 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:12.124001 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d4nqw" podUID="7209ee7f-3019-4465-8175-3317741f9b09" Jul 27 17:28:13.124504 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:13.124473 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-69zjf" Jul 27 17:28:13.124987 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:13.124603 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-69zjf" podUID="5115219e-8201-45d1-8a3f-6db85d5e741d" Jul 27 17:28:14.123924 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:14.123886 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d4nqw" Jul 27 17:28:14.124099 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:14.123890 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhzgt" Jul 27 17:28:14.124099 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:14.124007 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d4nqw" podUID="7209ee7f-3019-4465-8175-3317741f9b09" Jul 27 17:28:14.124099 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:14.124075 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhzgt" podUID="ec0c6ea9-94a4-41b6-9e1f-cccf978294ab" Jul 27 17:28:15.123496 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:15.123459 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-69zjf" Jul 27 17:28:15.123927 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:15.123604 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-69zjf" podUID="5115219e-8201-45d1-8a3f-6db85d5e741d" Jul 27 17:28:16.123954 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:16.123919 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhzgt" Jul 27 17:28:16.124344 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:16.123919 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d4nqw" Jul 27 17:28:16.124344 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:16.124039 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhzgt" podUID="ec0c6ea9-94a4-41b6-9e1f-cccf978294ab" Jul 27 17:28:16.124344 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:16.124141 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d4nqw" podUID="7209ee7f-3019-4465-8175-3317741f9b09" Jul 27 17:28:17.123537 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:17.123490 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-69zjf" Jul 27 17:28:17.123722 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:17.123653 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-69zjf" podUID="5115219e-8201-45d1-8a3f-6db85d5e741d" Jul 27 17:28:18.124015 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:18.123848 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d4nqw" Jul 27 17:28:18.124464 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:18.123883 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhzgt" Jul 27 17:28:18.124464 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:18.124084 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d4nqw" podUID="7209ee7f-3019-4465-8175-3317741f9b09" Jul 27 17:28:18.124464 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:18.124158 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhzgt" podUID="ec0c6ea9-94a4-41b6-9e1f-cccf978294ab" Jul 27 17:28:18.215236 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:18.215160 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4mbmr" event={"ID":"b9d10075-b019-4d6d-9904-bdccbc7e8ca1","Type":"ContainerStarted","Data":"aab3443b79ffbc1911b8c01a75bf3ddb82187f3718bee687963605b1a55a39bd"} Jul 27 17:28:18.216620 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:18.216592 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-185.ec2.internal" event={"ID":"92aa1923735291b61478f523669c7af8","Type":"ContainerStarted","Data":"4997e950284d72e4249eeeabcd14d113964ebb6eb9c062b9795f77b328590286"} Jul 27 17:28:18.217920 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:18.217897 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-w4d45" event={"ID":"44c736a9-a8db-4d86-8fb1-bb5287d8c319","Type":"ContainerStarted","Data":"6520b5f3dc3c018cbccf246595194dfb389b369271852b9400b45ae952319a19"} Jul 27 17:28:18.231600 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:18.231554 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-4mbmr" podStartSLOduration=1.7845216659999998 podStartE2EDuration="15.231540238s" totalImagesPullingTime="13.447018572s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:28:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:28:18.231012295 +0000 UTC m=+15.672227268" watchObservedRunningTime="2026-07-27 17:28:18.231540238 +0000 UTC m=+15.672755211" Jul 27 17:28:18.243934 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:18.243893 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-185.ec2.internal" podStartSLOduration=15.243884131 podStartE2EDuration="15.243884131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:28:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:28:18.243791923 +0000 UTC m=+15.685006897" watchObservedRunningTime="2026-07-27 17:28:18.243884131 +0000 UTC m=+15.685099105" Jul 27 17:28:18.261146 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:18.261104 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-w4d45" podStartSLOduration=1.825830235 podStartE2EDuration="15.261090333s" totalImagesPullingTime="13.435260098s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:28:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:28:18.260958538 +0000 UTC m=+15.702173523" watchObservedRunningTime="2026-07-27 17:28:18.261090333 +0000 UTC m=+15.702305308" Jul 27 17:28:19.123675 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:19.123503 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-69zjf" Jul 27 17:28:19.124257 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:19.123884 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-69zjf" podUID="5115219e-8201-45d1-8a3f-6db85d5e741d" Jul 27 17:28:19.222843 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:19.222110 2570 generic.go:356] "Generic (PLEG): container finished" podID="c1bf323d-120b-45ff-9117-f8142039ccfc" containerID="be178617aed9058e1d508644bfc4c7e3f200729c5f79e421f544752031396c8d" exitCode=0 Jul 27 17:28:19.222843 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:19.222201 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qp76t" event={"ID":"c1bf323d-120b-45ff-9117-f8142039ccfc","Type":"ContainerDied","Data":"be178617aed9058e1d508644bfc4c7e3f200729c5f79e421f544752031396c8d"} Jul 27 17:28:19.226615 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:19.226589 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-f5x5s" event={"ID":"acba0339-7147-4770-8e0b-ae7fff7eba7e","Type":"ContainerStarted","Data":"db361186cc2b4632d83bd078f0107ced941166ddadfd0e5a88ae471a14ed5309"} Jul 27 17:28:19.228232 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:19.228207 2570 generic.go:356] "Generic (PLEG): container finished" podID="96f53b3100180b7e57940a1d1a844aea" containerID="82b28fb440abb091ed7a5d438cdaca594aee9f493d03702e63a6687dc2b43a0d" exitCode=0 Jul 27 17:28:19.228346 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:19.228290 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-185.ec2.internal" event={"ID":"96f53b3100180b7e57940a1d1a844aea","Type":"ContainerDied","Data":"82b28fb440abb091ed7a5d438cdaca594aee9f493d03702e63a6687dc2b43a0d"} Jul 27 17:28:19.230477 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:19.230454 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-l5bjn" event={"ID":"0e66277b-cc74-421a-84d7-d1d43ce8de57","Type":"ContainerStarted","Data":"221ab91b7f5b1f9bcba7e1a108c1ae880313a338b1046b1afcfd7277151007fc"} Jul 27 17:28:19.232390 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:19.232363 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-f22lz" event={"ID":"b8d2f0e3-917e-4be9-adf5-e233e4f97048","Type":"ContainerStarted","Data":"a2e54f8002865759dc49e938943deebfe7c2d52b70fd638de3e04a8c5cdb82be"} Jul 27 17:28:19.234824 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:19.234801 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lnst" event={"ID":"3fd9dbff-b2d0-4c8b-a65a-8f9434f632c9","Type":"ContainerStarted","Data":"a02d891284d58f91be59f567e452903096abe957b0436adab8801c043ffa41d9"} Jul 27 17:28:19.266702 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:19.266653 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-f22lz" podStartSLOduration=2.871312686 podStartE2EDuration="16.266636148s" totalImagesPullingTime="13.395323462s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:28:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:28:19.266613024 +0000 UTC m=+16.707828002" watchObservedRunningTime="2026-07-27 17:28:19.266636148 +0000 UTC m=+16.707851321" Jul 27 17:28:19.279661 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:19.279606 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-f5x5s" podStartSLOduration=2.900186569 podStartE2EDuration="16.279585354s" totalImagesPullingTime="13.379398785s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:28:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:28:19.279550597 +0000 UTC m=+16.720765572" watchObservedRunningTime="2026-07-27 17:28:19.279585354 +0000 UTC m=+16.720800329" Jul 27 17:28:19.292421 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:19.292367 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-l5bjn" podStartSLOduration=2.941365439 podStartE2EDuration="16.292350861s" totalImagesPullingTime="13.350985422s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:28:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:28:19.292153889 +0000 UTC m=+16.733368863" watchObservedRunningTime="2026-07-27 17:28:19.292350861 +0000 UTC m=+16.733565850" Jul 27 17:28:19.664704 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:19.664493 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7209ee7f-3019-4465-8175-3317741f9b09-original-pull-secret\") pod \"global-pull-secret-syncer-d4nqw\" (UID: \"7209ee7f-3019-4465-8175-3317741f9b09\") " pod="kube-system/global-pull-secret-syncer-d4nqw" Jul 27 17:28:19.664886 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:19.664661 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 17:28:19.664886 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:19.664821 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7209ee7f-3019-4465-8175-3317741f9b09-original-pull-secret podName:7209ee7f-3019-4465-8175-3317741f9b09 nodeName:}" failed. No retries permitted until 2026-07-27 17:28:35.664799474 +0000 UTC m=+33.106014429 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7209ee7f-3019-4465-8175-3317741f9b09-original-pull-secret") pod "global-pull-secret-syncer-d4nqw" (UID: "7209ee7f-3019-4465-8175-3317741f9b09") : object "kube-system"/"original-pull-secret" not registered Jul 27 17:28:19.766065 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:19.766016 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5115219e-8201-45d1-8a3f-6db85d5e741d-metrics-certs\") pod \"network-metrics-daemon-69zjf\" (UID: \"5115219e-8201-45d1-8a3f-6db85d5e741d\") " pod="openshift-multus/network-metrics-daemon-69zjf" Jul 27 17:28:19.766249 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:19.766229 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 17:28:19.766315 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:19.766301 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5115219e-8201-45d1-8a3f-6db85d5e741d-metrics-certs podName:5115219e-8201-45d1-8a3f-6db85d5e741d nodeName:}" failed. No retries permitted until 2026-07-27 17:28:35.766282292 +0000 UTC m=+33.207497244 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5115219e-8201-45d1-8a3f-6db85d5e741d-metrics-certs") pod "network-metrics-daemon-69zjf" (UID: "5115219e-8201-45d1-8a3f-6db85d5e741d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 17:28:19.867303 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:19.867269 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p4xw\" (UniqueName: \"kubernetes.io/projected/ec0c6ea9-94a4-41b6-9e1f-cccf978294ab-kube-api-access-5p4xw\") pod \"network-check-target-jhzgt\" (UID: \"ec0c6ea9-94a4-41b6-9e1f-cccf978294ab\") " pod="openshift-network-diagnostics/network-check-target-jhzgt" Jul 27 17:28:19.867490 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:19.867467 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 17:28:19.867574 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:19.867494 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 17:28:19.867574 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:19.867508 2570 projected.go:196] Error preparing data for projected volume kube-api-access-5p4xw for pod openshift-network-diagnostics/network-check-target-jhzgt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 17:28:19.867733 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:19.867584 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec0c6ea9-94a4-41b6-9e1f-cccf978294ab-kube-api-access-5p4xw podName:ec0c6ea9-94a4-41b6-9e1f-cccf978294ab nodeName:}" failed. No retries permitted until 2026-07-27 17:28:35.867564134 +0000 UTC m=+33.308779090 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-5p4xw" (UniqueName: "kubernetes.io/projected/ec0c6ea9-94a4-41b6-9e1f-cccf978294ab-kube-api-access-5p4xw") pod "network-check-target-jhzgt" (UID: "ec0c6ea9-94a4-41b6-9e1f-cccf978294ab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 17:28:20.123914 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:20.123883 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhzgt" Jul 27 17:28:20.124073 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:20.123883 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d4nqw" Jul 27 17:28:20.124073 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:20.124017 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhzgt" podUID="ec0c6ea9-94a4-41b6-9e1f-cccf978294ab" Jul 27 17:28:20.124173 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:20.124123 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d4nqw" podUID="7209ee7f-3019-4465-8175-3317741f9b09" Jul 27 17:28:20.239183 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:20.239152 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8smpx" event={"ID":"9b6c93b0-6e89-41e6-accd-cbacae11cf4e","Type":"ContainerStarted","Data":"8f1b6c38914710f36e20dbfdda315cdeae3f29687a69f592abe03e518c28c414"} Jul 27 17:28:20.275889 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:20.275828 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-8smpx" podStartSLOduration=3.950881956 podStartE2EDuration="17.275807448s" totalImagesPullingTime="13.324925492s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:28:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:28:20.275787374 +0000 UTC m=+17.717002348" watchObservedRunningTime="2026-07-27 17:28:20.275807448 +0000 UTC m=+17.717022422" Jul 27 17:28:20.345963 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:20.345940 2570 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 27 17:28:20.729834 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:20.729793 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-l5bjn" Jul 27 17:28:20.998413 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:20.998235 2570 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-27T17:28:20.345959926Z","UUID":"0fead2c7-1ce4-4b5c-b2ad-fc7c079f70c2","Handler":null,"Name":"","Endpoint":""} Jul 27 17:28:21.000230 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:21.000206 2570 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 27 17:28:21.000365 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:21.000253 2570 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 27 17:28:21.123699 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:21.123664 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-69zjf" Jul 27 17:28:21.123873 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:21.123827 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-69zjf" podUID="5115219e-8201-45d1-8a3f-6db85d5e741d" Jul 27 17:28:21.243449 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:21.243381 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-185.ec2.internal" event={"ID":"96f53b3100180b7e57940a1d1a844aea","Type":"ContainerStarted","Data":"768a61ed88f1437d1dee6acff46d1c3e87803106851b91903608d43007d9582a"} Jul 27 17:28:21.245857 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:21.245822 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lnst" event={"ID":"3fd9dbff-b2d0-4c8b-a65a-8f9434f632c9","Type":"ContainerStarted","Data":"624db8aaef55014edc084fd50bdfcfb388e2d2da5246f1990ea9dab3efd2b1e3"} Jul 27 17:28:21.258888 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:21.258834 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-185.ec2.internal" podStartSLOduration=18.258815377 podStartE2EDuration="18.258815377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:28:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:28:21.258657206 +0000 UTC m=+18.699872184" watchObservedRunningTime="2026-07-27 17:28:21.258815377 +0000 UTC m=+18.700030354" Jul 27 17:28:22.123515 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:22.123476 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d4nqw" Jul 27 17:28:22.123710 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:22.123539 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhzgt" Jul 27 17:28:22.123710 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:22.123632 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d4nqw" podUID="7209ee7f-3019-4465-8175-3317741f9b09" Jul 27 17:28:22.123804 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:22.123759 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhzgt" podUID="ec0c6ea9-94a4-41b6-9e1f-cccf978294ab" Jul 27 17:28:22.174948 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:22.174920 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-l5bjn" Jul 27 17:28:22.175485 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:22.175461 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-l5bjn" Jul 27 17:28:22.253863 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:22.253648 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-l5bjn" Jul 27 17:28:23.127830 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:23.125044 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-69zjf" Jul 27 17:28:23.127830 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:23.125396 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-69zjf" podUID="5115219e-8201-45d1-8a3f-6db85d5e741d" Jul 27 17:28:23.256237 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:23.256194 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lnst" event={"ID":"3fd9dbff-b2d0-4c8b-a65a-8f9434f632c9","Type":"ContainerStarted","Data":"2d7b184b4da2121091afeb0f84811b7630a827167f93511633236210d762a5ac"} Jul 27 17:28:23.274414 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:23.274353 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lnst" podStartSLOduration=3.314097225 podStartE2EDuration="20.274332422s" totalImagesPullingTime="16.960235197s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:28:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:28:23.273722846 +0000 UTC m=+20.714937821" watchObservedRunningTime="2026-07-27 17:28:23.274332422 +0000 UTC m=+20.715547397" Jul 27 17:28:24.123747 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:24.123484 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhzgt" Jul 27 17:28:24.123921 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:24.123484 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d4nqw" Jul 27 17:28:24.123921 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:24.123858 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhzgt" podUID="ec0c6ea9-94a4-41b6-9e1f-cccf978294ab" Jul 27 17:28:24.124042 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:24.123935 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d4nqw" podUID="7209ee7f-3019-4465-8175-3317741f9b09" Jul 27 17:28:25.124210 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:25.124174 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-69zjf" Jul 27 17:28:25.124804 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:25.124303 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-69zjf" podUID="5115219e-8201-45d1-8a3f-6db85d5e741d" Jul 27 17:28:25.262458 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:25.262420 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qp76t" event={"ID":"c1bf323d-120b-45ff-9117-f8142039ccfc","Type":"ContainerStarted","Data":"14f365acdc33619a381ed1590452deb849a601c7c1822d2bfdaa0a0a37a24d2a"} Jul 27 17:28:26.123712 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:26.123673 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d4nqw" Jul 27 17:28:26.123985 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:26.123728 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhzgt" Jul 27 17:28:26.123985 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:26.123811 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d4nqw" podUID="7209ee7f-3019-4465-8175-3317741f9b09" Jul 27 17:28:26.123985 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:26.123950 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhzgt" podUID="ec0c6ea9-94a4-41b6-9e1f-cccf978294ab" Jul 27 17:28:27.124139 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:27.124101 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-69zjf" Jul 27 17:28:27.124587 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:27.124244 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-69zjf" podUID="5115219e-8201-45d1-8a3f-6db85d5e741d" Jul 27 17:28:28.124371 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:28.124331 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d4nqw" Jul 27 17:28:28.124371 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:28.124366 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhzgt" Jul 27 17:28:28.124870 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:28.124478 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d4nqw" podUID="7209ee7f-3019-4465-8175-3317741f9b09" Jul 27 17:28:28.124870 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:28.124626 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhzgt" podUID="ec0c6ea9-94a4-41b6-9e1f-cccf978294ab" Jul 27 17:28:29.123975 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:29.123948 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-69zjf" Jul 27 17:28:29.124153 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:29.124085 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-69zjf" podUID="5115219e-8201-45d1-8a3f-6db85d5e741d" Jul 27 17:28:29.272267 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:29.272234 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" event={"ID":"4dbb97f8-c6bc-44e9-b468-e3577bb468af","Type":"ContainerStarted","Data":"c78fc33fb54e7ae46266e7eec3fdd11b06688ac4e87802368ecf41c733f58686"} Jul 27 17:28:29.272674 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:29.272308 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" event={"ID":"4dbb97f8-c6bc-44e9-b468-e3577bb468af","Type":"ContainerStarted","Data":"e66c17c4cb944f5a02914632fb1e97458d3449aa4c02991316b5911dc8c632e2"} Jul 27 17:28:29.272674 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:29.272355 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" event={"ID":"4dbb97f8-c6bc-44e9-b468-e3577bb468af","Type":"ContainerStarted","Data":"8fbfae8883213cbb471f7fc29be92ac7b161ddd9e24612e08423123e04fd99e2"} Jul 27 17:28:29.272674 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:29.272392 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" event={"ID":"4dbb97f8-c6bc-44e9-b468-e3577bb468af","Type":"ContainerStarted","Data":"fabcf75a86785f82ed79a7f9bae829f9dc65154f7b088141b49e5039bd156c19"} Jul 27 17:28:29.274149 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:29.274125 2570 generic.go:356] "Generic (PLEG): container finished" podID="c1bf323d-120b-45ff-9117-f8142039ccfc" containerID="14f365acdc33619a381ed1590452deb849a601c7c1822d2bfdaa0a0a37a24d2a" exitCode=0 Jul 27 17:28:29.274236 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:29.274161 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qp76t" event={"ID":"c1bf323d-120b-45ff-9117-f8142039ccfc","Type":"ContainerDied","Data":"14f365acdc33619a381ed1590452deb849a601c7c1822d2bfdaa0a0a37a24d2a"} Jul 27 17:28:30.123891 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:30.123684 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d4nqw" Jul 27 17:28:30.124063 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:30.123736 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhzgt" Jul 27 17:28:30.124063 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:30.124001 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d4nqw" podUID="7209ee7f-3019-4465-8175-3317741f9b09" Jul 27 17:28:30.124063 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:30.124036 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhzgt" podUID="ec0c6ea9-94a4-41b6-9e1f-cccf978294ab" Jul 27 17:28:30.278489 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:30.278451 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" event={"ID":"4dbb97f8-c6bc-44e9-b468-e3577bb468af","Type":"ContainerStarted","Data":"4c59bc1eedf16654f0613175fa5aad312287648c7578a4472c67cdb488a98d98"} Jul 27 17:28:30.278953 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:30.278589 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" event={"ID":"4dbb97f8-c6bc-44e9-b468-e3577bb468af","Type":"ContainerStarted","Data":"51351c3e52cce831b7b5fd33d18ee1c65a90117b585963f7edcc02c3d30dbee0"} Jul 27 17:28:31.123845 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:31.123813 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-69zjf" Jul 27 17:28:31.124009 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:31.123929 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-69zjf" podUID="5115219e-8201-45d1-8a3f-6db85d5e741d" Jul 27 17:28:31.282861 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:31.282830 2570 generic.go:356] "Generic (PLEG): container finished" podID="c1bf323d-120b-45ff-9117-f8142039ccfc" containerID="819e74ef42a1b643bce52b6bddc54d09c2f6816a388c640f7281c21f1dadef9f" exitCode=0 Jul 27 17:28:31.283303 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:31.282870 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qp76t" event={"ID":"c1bf323d-120b-45ff-9117-f8142039ccfc","Type":"ContainerDied","Data":"819e74ef42a1b643bce52b6bddc54d09c2f6816a388c640f7281c21f1dadef9f"} Jul 27 17:28:32.123642 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:32.123611 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d4nqw" Jul 27 17:28:32.123835 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:32.123616 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhzgt" Jul 27 17:28:32.123835 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:32.123726 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d4nqw" podUID="7209ee7f-3019-4465-8175-3317741f9b09" Jul 27 17:28:32.123835 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:32.123778 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhzgt" podUID="ec0c6ea9-94a4-41b6-9e1f-cccf978294ab" Jul 27 17:28:32.289626 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:32.289587 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" event={"ID":"4dbb97f8-c6bc-44e9-b468-e3577bb468af","Type":"ContainerStarted","Data":"9876a6b2836572f097accc65c8c5ce455e37018ccbc77fc716640acfd7b8fce5"} Jul 27 17:28:33.124302 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:33.124265 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-69zjf" Jul 27 17:28:33.124471 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:33.124363 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-69zjf" podUID="5115219e-8201-45d1-8a3f-6db85d5e741d" Jul 27 17:28:34.123715 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:34.123678 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhzgt" Jul 27 17:28:34.124425 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:34.123678 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d4nqw" Jul 27 17:28:34.124425 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:34.123807 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhzgt" podUID="ec0c6ea9-94a4-41b6-9e1f-cccf978294ab" Jul 27 17:28:34.124425 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:34.123857 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d4nqw" podUID="7209ee7f-3019-4465-8175-3317741f9b09" Jul 27 17:28:34.297888 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:34.297684 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" event={"ID":"4dbb97f8-c6bc-44e9-b468-e3577bb468af","Type":"ContainerStarted","Data":"9776f6d3ebc63e97d9e0e9309ef9c49d7b26bef6984a070f1d015e86aaefb5bc"} Jul 27 17:28:34.298053 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:34.298020 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" Jul 27 17:28:34.298121 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:34.298108 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" Jul 27 17:28:34.300161 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:34.300138 2570 generic.go:356] "Generic (PLEG): container finished" podID="c1bf323d-120b-45ff-9117-f8142039ccfc" containerID="ec3d8b547a0979becc20117dbb76b1de23e6a561b12606076fa45b8c4bc01e28" exitCode=0 Jul 27 17:28:34.300263 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:34.300171 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qp76t" event={"ID":"c1bf323d-120b-45ff-9117-f8142039ccfc","Type":"ContainerDied","Data":"ec3d8b547a0979becc20117dbb76b1de23e6a561b12606076fa45b8c4bc01e28"} Jul 27 17:28:34.314491 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:34.314452 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" Jul 27 17:28:34.326926 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:34.326884 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" podStartSLOduration=6.860141282 podStartE2EDuration="31.326868861s" totalImagesPullingTime="24.466727579s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:28:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=2 observedRunningTime="2026-07-27 17:28:34.326756505 +0000 UTC m=+31.767971478" watchObservedRunningTime="2026-07-27 17:28:34.326868861 +0000 UTC m=+31.768083835" Jul 27 17:28:35.123615 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:35.123581 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-69zjf" Jul 27 17:28:35.123788 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:35.123711 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-69zjf" podUID="5115219e-8201-45d1-8a3f-6db85d5e741d" Jul 27 17:28:35.303287 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:35.303249 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" Jul 27 17:28:35.325106 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:35.325077 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" Jul 27 17:28:35.690855 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:35.690822 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7209ee7f-3019-4465-8175-3317741f9b09-original-pull-secret\") pod \"global-pull-secret-syncer-d4nqw\" (UID: \"7209ee7f-3019-4465-8175-3317741f9b09\") " pod="kube-system/global-pull-secret-syncer-d4nqw" Jul 27 17:28:35.690985 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:35.690963 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 17:28:35.691042 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:35.691034 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7209ee7f-3019-4465-8175-3317741f9b09-original-pull-secret podName:7209ee7f-3019-4465-8175-3317741f9b09 nodeName:}" failed. No retries permitted until 2026-07-27 17:29:07.691013051 +0000 UTC m=+65.132228004 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7209ee7f-3019-4465-8175-3317741f9b09-original-pull-secret") pod "global-pull-secret-syncer-d4nqw" (UID: "7209ee7f-3019-4465-8175-3317741f9b09") : object "kube-system"/"original-pull-secret" not registered Jul 27 17:28:35.791196 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:35.791166 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5115219e-8201-45d1-8a3f-6db85d5e741d-metrics-certs\") pod \"network-metrics-daemon-69zjf\" (UID: \"5115219e-8201-45d1-8a3f-6db85d5e741d\") " pod="openshift-multus/network-metrics-daemon-69zjf" Jul 27 17:28:35.791349 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:35.791333 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 17:28:35.791419 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:35.791408 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5115219e-8201-45d1-8a3f-6db85d5e741d-metrics-certs podName:5115219e-8201-45d1-8a3f-6db85d5e741d nodeName:}" failed. No retries permitted until 2026-07-27 17:29:07.791387412 +0000 UTC m=+65.232602364 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5115219e-8201-45d1-8a3f-6db85d5e741d-metrics-certs") pod "network-metrics-daemon-69zjf" (UID: "5115219e-8201-45d1-8a3f-6db85d5e741d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 17:28:35.888741 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:35.888549 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jhzgt"] Jul 27 17:28:35.888741 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:35.888677 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhzgt" Jul 27 17:28:35.889014 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:35.888839 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhzgt" podUID="ec0c6ea9-94a4-41b6-9e1f-cccf978294ab" Jul 27 17:28:35.889451 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:35.889409 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-69zjf"] Jul 27 17:28:35.889575 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:35.889550 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-69zjf" Jul 27 17:28:35.889682 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:35.889659 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-69zjf" podUID="5115219e-8201-45d1-8a3f-6db85d5e741d" Jul 27 17:28:35.890423 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:35.890356 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-d4nqw"] Jul 27 17:28:35.890552 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:35.890448 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d4nqw" Jul 27 17:28:35.890626 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:35.890544 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d4nqw" podUID="7209ee7f-3019-4465-8175-3317741f9b09" Jul 27 17:28:35.892168 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:35.892055 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p4xw\" (UniqueName: \"kubernetes.io/projected/ec0c6ea9-94a4-41b6-9e1f-cccf978294ab-kube-api-access-5p4xw\") pod \"network-check-target-jhzgt\" (UID: \"ec0c6ea9-94a4-41b6-9e1f-cccf978294ab\") " pod="openshift-network-diagnostics/network-check-target-jhzgt" Jul 27 17:28:35.892256 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:35.892202 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 17:28:35.892256 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:35.892219 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 17:28:35.892256 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:35.892231 2570 projected.go:196] Error preparing data for projected volume kube-api-access-5p4xw for pod openshift-network-diagnostics/network-check-target-jhzgt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 17:28:35.892424 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:35.892284 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec0c6ea9-94a4-41b6-9e1f-cccf978294ab-kube-api-access-5p4xw podName:ec0c6ea9-94a4-41b6-9e1f-cccf978294ab nodeName:}" failed. No retries permitted until 2026-07-27 17:29:07.892266959 +0000 UTC m=+65.333481913 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-5p4xw" (UniqueName: "kubernetes.io/projected/ec0c6ea9-94a4-41b6-9e1f-cccf978294ab-kube-api-access-5p4xw") pod "network-check-target-jhzgt" (UID: "ec0c6ea9-94a4-41b6-9e1f-cccf978294ab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 17:28:37.124113 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:37.124082 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhzgt" Jul 27 17:28:37.124638 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:37.124220 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhzgt" podUID="ec0c6ea9-94a4-41b6-9e1f-cccf978294ab" Jul 27 17:28:38.123793 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:38.123764 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d4nqw" Jul 27 17:28:38.123961 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:38.123768 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-69zjf" Jul 27 17:28:38.123961 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:38.123879 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d4nqw" podUID="7209ee7f-3019-4465-8175-3317741f9b09" Jul 27 17:28:38.124077 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:38.123979 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-69zjf" podUID="5115219e-8201-45d1-8a3f-6db85d5e741d" Jul 27 17:28:39.123582 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:39.123369 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhzgt" Jul 27 17:28:39.123958 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:39.123668 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhzgt" podUID="ec0c6ea9-94a4-41b6-9e1f-cccf978294ab" Jul 27 17:28:39.313723 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:39.313694 2570 generic.go:356] "Generic (PLEG): container finished" podID="c1bf323d-120b-45ff-9117-f8142039ccfc" containerID="eebea1b2ca64b555c9bd61f3e597c0596650b379ba110ac231e54c85019b254a" exitCode=0 Jul 27 17:28:39.313881 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:39.313735 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qp76t" event={"ID":"c1bf323d-120b-45ff-9117-f8142039ccfc","Type":"ContainerDied","Data":"eebea1b2ca64b555c9bd61f3e597c0596650b379ba110ac231e54c85019b254a"} Jul 27 17:28:40.123469 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:40.123432 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-69zjf" Jul 27 17:28:40.123469 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:40.123454 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d4nqw" Jul 27 17:28:40.123961 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:40.123583 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-69zjf" podUID="5115219e-8201-45d1-8a3f-6db85d5e741d" Jul 27 17:28:40.123961 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:40.123663 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d4nqw" podUID="7209ee7f-3019-4465-8175-3317741f9b09" Jul 27 17:28:40.318963 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:40.318925 2570 generic.go:356] "Generic (PLEG): container finished" podID="c1bf323d-120b-45ff-9117-f8142039ccfc" containerID="bf5b4977a1c46e95161cf9091f358d0090444f0eaa3055258aca49458c78d301" exitCode=0 Jul 27 17:28:40.318963 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:40.318965 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qp76t" event={"ID":"c1bf323d-120b-45ff-9117-f8142039ccfc","Type":"ContainerDied","Data":"bf5b4977a1c46e95161cf9091f358d0090444f0eaa3055258aca49458c78d301"} Jul 27 17:28:41.123569 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:41.123534 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhzgt" Jul 27 17:28:41.123742 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:41.123642 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhzgt" podUID="ec0c6ea9-94a4-41b6-9e1f-cccf978294ab" Jul 27 17:28:41.324285 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:41.324254 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qp76t" event={"ID":"c1bf323d-120b-45ff-9117-f8142039ccfc","Type":"ContainerStarted","Data":"70c8ef90823ae76f39ce91c92cf58e923b6f6622acb5c19f01c149d54b3ed864"} Jul 27 17:28:41.354383 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:41.354333 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-qp76t" podStartSLOduration=11.947908226 podStartE2EDuration="38.354318445s" totalImagesPullingTime="26.406410219s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:28:03 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:28:41.352570771 +0000 UTC m=+38.793785744" watchObservedRunningTime="2026-07-27 17:28:41.354318445 +0000 UTC m=+38.795533419" Jul 27 17:28:41.875490 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:41.875420 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-185.ec2.internal" event="NodeReady" Jul 27 17:28:41.875644 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:41.875556 2570 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 27 17:28:41.952301 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:41.952269 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-dpkfg"] Jul 27 17:28:41.978652 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:41.978619 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-gc5j9"] Jul 27 17:28:41.978795 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:41.978753 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dpkfg" Jul 27 17:28:41.982182 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:41.982161 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 27 17:28:41.982182 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:41.982173 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-x9hw5\"" Jul 27 17:28:41.982353 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:41.982174 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 27 17:28:42.001908 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:42.001881 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gc5j9" Jul 27 17:28:42.006332 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:42.006311 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 27 17:28:42.007071 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:42.007054 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-c5l2l\"" Jul 27 17:28:42.007173 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:42.007108 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 27 17:28:42.007428 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:42.007399 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 27 17:28:42.012433 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:42.012412 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gc5j9"] Jul 27 17:28:42.018228 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:42.018208 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dpkfg"] Jul 27 17:28:42.018709 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:42.018694 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-rzskr"] Jul 27 17:28:42.038020 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:42.037995 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rzskr" Jul 27 17:28:42.044551 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:42.044511 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vzxqf\"" Jul 27 17:28:42.047540 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:42.047510 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 27 17:28:42.047601 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:42.047547 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 27 17:28:42.047601 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:42.047557 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 27 17:28:42.047601 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:42.047587 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 27 17:28:42.053864 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:42.053843 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rzskr"] Jul 27 17:28:42.068837 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:42.068818 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-f22lz_b8d2f0e3-917e-4be9-adf5-e233e4f97048/dns-node-resolver/0.log" Jul 27 17:28:42.124292 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:42.124262 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-69zjf" Jul 27 17:28:42.124292 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:42.124290 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d4nqw" Jul 27 17:28:42.127289 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:42.127238 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6sz2v\"" Jul 27 17:28:42.127380 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:42.127294 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 27 17:28:42.127659 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:42.127646 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 27 17:28:42.141198 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:42.141182 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhzm7\" (UniqueName: \"kubernetes.io/projected/91b9ae5b-5c22-41ab-8845-cd9f65146cbc-kube-api-access-zhzm7\") pod \"dns-default-dpkfg\" (UID: \"91b9ae5b-5c22-41ab-8845-cd9f65146cbc\") " pod="openshift-dns/dns-default-dpkfg" Jul 27 17:28:42.141279 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:42.141212 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mc6s\" (UniqueName: \"kubernetes.io/projected/1596f129-a503-4513-bb1f-3b7ac6339c51-kube-api-access-8mc6s\") pod \"ingress-canary-gc5j9\" (UID: \"1596f129-a503-4513-bb1f-3b7ac6339c51\") " pod="openshift-ingress-canary/ingress-canary-gc5j9" Jul 27 17:28:42.141279 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:42.141231 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7e90d28e-739e-4b37-acdb-6997186a73e8-crio-socket\") pod \"insights-runtime-extractor-rzskr\" (UID: \"7e90d28e-739e-4b37-acdb-6997186a73e8\") " pod="openshift-insights/insights-runtime-extractor-rzskr" Jul 27 17:28:42.141279 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:42.141249 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91b9ae5b-5c22-41ab-8845-cd9f65146cbc-tmp-dir\") pod \"dns-default-dpkfg\" (UID: \"91b9ae5b-5c22-41ab-8845-cd9f65146cbc\") " pod="openshift-dns/dns-default-dpkfg" Jul 27 17:28:42.141279 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:42.141273 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7e90d28e-739e-4b37-acdb-6997186a73e8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rzskr\" (UID: \"7e90d28e-739e-4b37-acdb-6997186a73e8\") " pod="openshift-insights/insights-runtime-extractor-rzskr" Jul 27 17:28:42.141408 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:42.141331 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7e90d28e-739e-4b37-acdb-6997186a73e8-data-volume\") pod \"insights-runtime-extractor-rzskr\" (UID: \"7e90d28e-739e-4b37-acdb-6997186a73e8\") " pod="openshift-insights/insights-runtime-extractor-rzskr" Jul 27 17:28:42.141408 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:42.141352 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7e90d28e-739e-4b37-acdb-6997186a73e8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rzskr\" (UID: \"7e90d28e-739e-4b37-acdb-6997186a73e8\") " pod="openshift-insights/insights-runtime-extractor-rzskr" Jul 27 17:28:42.141408 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:42.141373 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz9wd\" (UniqueName: \"kubernetes.io/projected/7e90d28e-739e-4b37-acdb-6997186a73e8-kube-api-access-qz9wd\") pod \"insights-runtime-extractor-rzskr\" (UID: \"7e90d28e-739e-4b37-acdb-6997186a73e8\") " pod="openshift-insights/insights-runtime-extractor-rzskr" Jul 27 17:28:42.141408 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:42.141398 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/91b9ae5b-5c22-41ab-8845-cd9f65146cbc-config-volume\") pod \"dns-default-dpkfg\" (UID: \"91b9ae5b-5c22-41ab-8845-cd9f65146cbc\") " pod="openshift-dns/dns-default-dpkfg" Jul 27 17:28:42.141552 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:42.141413 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/91b9ae5b-5c22-41ab-8845-cd9f65146cbc-metrics-tls\") pod \"dns-default-dpkfg\" (UID: \"91b9ae5b-5c22-41ab-8845-cd9f65146cbc\") " pod="openshift-dns/dns-default-dpkfg" Jul 27 17:28:42.141552 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:42.141460 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1596f129-a503-4513-bb1f-3b7ac6339c51-cert\") pod \"ingress-canary-gc5j9\" (UID: \"1596f129-a503-4513-bb1f-3b7ac6339c51\") " pod="openshift-ingress-canary/ingress-canary-gc5j9" Jul 27 17:28:42.242707 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:42.242667 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1596f129-a503-4513-bb1f-3b7ac6339c51-cert\") pod \"ingress-canary-gc5j9\" (UID: \"1596f129-a503-4513-bb1f-3b7ac6339c51\") " pod="openshift-ingress-canary/ingress-canary-gc5j9" Jul 27 17:28:42.242707 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:42.242713 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhzm7\" (UniqueName: \"kubernetes.io/projected/91b9ae5b-5c22-41ab-8845-cd9f65146cbc-kube-api-access-zhzm7\") pod \"dns-default-dpkfg\" (UID: \"91b9ae5b-5c22-41ab-8845-cd9f65146cbc\") " pod="openshift-dns/dns-default-dpkfg" Jul 27 17:28:42.242975 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:42.242834 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mc6s\" (UniqueName: \"kubernetes.io/projected/1596f129-a503-4513-bb1f-3b7ac6339c51-kube-api-access-8mc6s\") pod \"ingress-canary-gc5j9\" (UID: \"1596f129-a503-4513-bb1f-3b7ac6339c51\") " pod="openshift-ingress-canary/ingress-canary-gc5j9" Jul 27 17:28:42.242975 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:42.242867 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7e90d28e-739e-4b37-acdb-6997186a73e8-crio-socket\") pod \"insights-runtime-extractor-rzskr\" (UID: \"7e90d28e-739e-4b37-acdb-6997186a73e8\") " pod="openshift-insights/insights-runtime-extractor-rzskr" Jul 27 17:28:42.242975 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:42.242900 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91b9ae5b-5c22-41ab-8845-cd9f65146cbc-tmp-dir\") pod \"dns-default-dpkfg\" (UID: \"91b9ae5b-5c22-41ab-8845-cd9f65146cbc\") " pod="openshift-dns/dns-default-dpkfg" Jul 27 17:28:42.243123 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:42.243001 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7e90d28e-739e-4b37-acdb-6997186a73e8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rzskr\" (UID: \"7e90d28e-739e-4b37-acdb-6997186a73e8\") " pod="openshift-insights/insights-runtime-extractor-rzskr" Jul 27 17:28:42.243123 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:42.243082 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7e90d28e-739e-4b37-acdb-6997186a73e8-data-volume\") pod \"insights-runtime-extractor-rzskr\" (UID: \"7e90d28e-739e-4b37-acdb-6997186a73e8\") " pod="openshift-insights/insights-runtime-extractor-rzskr" Jul 27 17:28:42.243254 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:42.243106 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7e90d28e-739e-4b37-acdb-6997186a73e8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rzskr\" (UID: \"7e90d28e-739e-4b37-acdb-6997186a73e8\") " pod="openshift-insights/insights-runtime-extractor-rzskr" Jul 27 17:28:42.243254 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:42.243117 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7e90d28e-739e-4b37-acdb-6997186a73e8-crio-socket\") pod \"insights-runtime-extractor-rzskr\" (UID: \"7e90d28e-739e-4b37-acdb-6997186a73e8\") " pod="openshift-insights/insights-runtime-extractor-rzskr" Jul 27 17:28:42.243254 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:42.243162 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 27 17:28:42.243254 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:42.243191 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91b9ae5b-5c22-41ab-8845-cd9f65146cbc-tmp-dir\") pod \"dns-default-dpkfg\" (UID: \"91b9ae5b-5c22-41ab-8845-cd9f65146cbc\") " pod="openshift-dns/dns-default-dpkfg" Jul 27 17:28:42.243254 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:42.243201 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 17:28:42.243254 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:42.243227 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz9wd\" (UniqueName: \"kubernetes.io/projected/7e90d28e-739e-4b37-acdb-6997186a73e8-kube-api-access-qz9wd\") pod \"insights-runtime-extractor-rzskr\" (UID: \"7e90d28e-739e-4b37-acdb-6997186a73e8\") " pod="openshift-insights/insights-runtime-extractor-rzskr" Jul 27 17:28:42.243254 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:42.243253 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7e90d28e-739e-4b37-acdb-6997186a73e8-kube-rbac-proxy-cm podName:7e90d28e-739e-4b37-acdb-6997186a73e8 nodeName:}" failed. No retries permitted until 2026-07-27 17:28:42.743232343 +0000 UTC m=+40.184447315 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7e90d28e-739e-4b37-acdb-6997186a73e8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rzskr" (UID: "7e90d28e-739e-4b37-acdb-6997186a73e8") : configmap "kube-rbac-proxy" not found Jul 27 17:28:42.243621 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:42.243291 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e90d28e-739e-4b37-acdb-6997186a73e8-insights-runtime-extractor-tls podName:7e90d28e-739e-4b37-acdb-6997186a73e8 nodeName:}" failed. No retries permitted until 2026-07-27 17:28:42.743274159 +0000 UTC m=+40.184489117 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7e90d28e-739e-4b37-acdb-6997186a73e8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rzskr" (UID: "7e90d28e-739e-4b37-acdb-6997186a73e8") : secret "insights-runtime-extractor-tls" not found Jul 27 17:28:42.243621 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:42.243330 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/91b9ae5b-5c22-41ab-8845-cd9f65146cbc-config-volume\") pod \"dns-default-dpkfg\" (UID: \"91b9ae5b-5c22-41ab-8845-cd9f65146cbc\") " pod="openshift-dns/dns-default-dpkfg" Jul 27 17:28:42.243621 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:42.243354 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/91b9ae5b-5c22-41ab-8845-cd9f65146cbc-metrics-tls\") pod \"dns-default-dpkfg\" (UID: \"91b9ae5b-5c22-41ab-8845-cd9f65146cbc\") " pod="openshift-dns/dns-default-dpkfg" Jul 27 17:28:42.243621 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:42.243408 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7e90d28e-739e-4b37-acdb-6997186a73e8-data-volume\") pod \"insights-runtime-extractor-rzskr\" (UID: \"7e90d28e-739e-4b37-acdb-6997186a73e8\") " pod="openshift-insights/insights-runtime-extractor-rzskr" Jul 27 17:28:42.243621 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:42.243505 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 17:28:42.243621 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:42.243557 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91b9ae5b-5c22-41ab-8845-cd9f65146cbc-metrics-tls podName:91b9ae5b-5c22-41ab-8845-cd9f65146cbc nodeName:}" failed. No retries permitted until 2026-07-27 17:28:42.743548679 +0000 UTC m=+40.184763631 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/91b9ae5b-5c22-41ab-8845-cd9f65146cbc-metrics-tls") pod "dns-default-dpkfg" (UID: "91b9ae5b-5c22-41ab-8845-cd9f65146cbc") : secret "dns-default-metrics-tls" not found Jul 27 17:28:42.243806 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:42.243745 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/91b9ae5b-5c22-41ab-8845-cd9f65146cbc-config-volume\") pod \"dns-default-dpkfg\" (UID: \"91b9ae5b-5c22-41ab-8845-cd9f65146cbc\") " pod="openshift-dns/dns-default-dpkfg" Jul 27 17:28:42.246909 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:42.246892 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1596f129-a503-4513-bb1f-3b7ac6339c51-cert\") pod \"ingress-canary-gc5j9\" (UID: \"1596f129-a503-4513-bb1f-3b7ac6339c51\") " pod="openshift-ingress-canary/ingress-canary-gc5j9" Jul 27 17:28:42.256978 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:42.256512 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mc6s\" (UniqueName: \"kubernetes.io/projected/1596f129-a503-4513-bb1f-3b7ac6339c51-kube-api-access-8mc6s\") pod \"ingress-canary-gc5j9\" (UID: \"1596f129-a503-4513-bb1f-3b7ac6339c51\") " pod="openshift-ingress-canary/ingress-canary-gc5j9" Jul 27 17:28:42.256978 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:42.256884 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhzm7\" (UniqueName: \"kubernetes.io/projected/91b9ae5b-5c22-41ab-8845-cd9f65146cbc-kube-api-access-zhzm7\") pod \"dns-default-dpkfg\" (UID: \"91b9ae5b-5c22-41ab-8845-cd9f65146cbc\") " pod="openshift-dns/dns-default-dpkfg" Jul 27 17:28:42.276663 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:42.276633 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz9wd\" (UniqueName: \"kubernetes.io/projected/7e90d28e-739e-4b37-acdb-6997186a73e8-kube-api-access-qz9wd\") pod \"insights-runtime-extractor-rzskr\" (UID: \"7e90d28e-739e-4b37-acdb-6997186a73e8\") " pod="openshift-insights/insights-runtime-extractor-rzskr" Jul 27 17:28:42.310243 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:42.310203 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gc5j9" Jul 27 17:28:42.517055 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:42.517022 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gc5j9"] Jul 27 17:28:42.520804 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:42.520779 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1596f129_a503_4513_bb1f_3b7ac6339c51.slice/crio-cf043ad5ffd5473488d1d3089b50d455a7b07dede12e10585cef929ee3294631 WatchSource:0}: Error finding container cf043ad5ffd5473488d1d3089b50d455a7b07dede12e10585cef929ee3294631: Status 404 returned error can't find the container with id cf043ad5ffd5473488d1d3089b50d455a7b07dede12e10585cef929ee3294631 Jul 27 17:28:42.748339 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:42.748251 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/91b9ae5b-5c22-41ab-8845-cd9f65146cbc-metrics-tls\") pod \"dns-default-dpkfg\" (UID: \"91b9ae5b-5c22-41ab-8845-cd9f65146cbc\") " pod="openshift-dns/dns-default-dpkfg" Jul 27 17:28:42.748339 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:42.748332 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7e90d28e-739e-4b37-acdb-6997186a73e8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rzskr\" (UID: \"7e90d28e-739e-4b37-acdb-6997186a73e8\") " pod="openshift-insights/insights-runtime-extractor-rzskr" Jul 27 17:28:42.748590 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:42.748364 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7e90d28e-739e-4b37-acdb-6997186a73e8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rzskr\" (UID: \"7e90d28e-739e-4b37-acdb-6997186a73e8\") " pod="openshift-insights/insights-runtime-extractor-rzskr" Jul 27 17:28:42.748590 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:42.748390 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 17:28:42.748590 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:42.748445 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 27 17:28:42.748590 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:42.748454 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91b9ae5b-5c22-41ab-8845-cd9f65146cbc-metrics-tls podName:91b9ae5b-5c22-41ab-8845-cd9f65146cbc nodeName:}" failed. No retries permitted until 2026-07-27 17:28:43.748439257 +0000 UTC m=+41.189654210 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/91b9ae5b-5c22-41ab-8845-cd9f65146cbc-metrics-tls") pod "dns-default-dpkfg" (UID: "91b9ae5b-5c22-41ab-8845-cd9f65146cbc") : secret "dns-default-metrics-tls" not found Jul 27 17:28:42.748590 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:42.748464 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 17:28:42.748590 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:42.748498 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7e90d28e-739e-4b37-acdb-6997186a73e8-kube-rbac-proxy-cm podName:7e90d28e-739e-4b37-acdb-6997186a73e8 nodeName:}" failed. No retries permitted until 2026-07-27 17:28:43.748481151 +0000 UTC m=+41.189696122 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7e90d28e-739e-4b37-acdb-6997186a73e8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rzskr" (UID: "7e90d28e-739e-4b37-acdb-6997186a73e8") : configmap "kube-rbac-proxy" not found Jul 27 17:28:42.748590 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:42.748514 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e90d28e-739e-4b37-acdb-6997186a73e8-insights-runtime-extractor-tls podName:7e90d28e-739e-4b37-acdb-6997186a73e8 nodeName:}" failed. No retries permitted until 2026-07-27 17:28:43.748505434 +0000 UTC m=+41.189720391 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7e90d28e-739e-4b37-acdb-6997186a73e8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rzskr" (UID: "7e90d28e-739e-4b37-acdb-6997186a73e8") : secret "insights-runtime-extractor-tls" not found Jul 27 17:28:42.769613 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:42.769582 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-f5x5s_acba0339-7147-4770-8e0b-ae7fff7eba7e/node-ca/0.log" Jul 27 17:28:43.125690 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:43.125418 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhzgt" Jul 27 17:28:43.130577 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:43.130554 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 27 17:28:43.130904 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:43.130884 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 27 17:28:43.131402 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:43.131383 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kcn4g\"" Jul 27 17:28:43.330730 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:43.330702 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gc5j9" event={"ID":"1596f129-a503-4513-bb1f-3b7ac6339c51","Type":"ContainerStarted","Data":"cf043ad5ffd5473488d1d3089b50d455a7b07dede12e10585cef929ee3294631"} Jul 27 17:28:43.755804 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:43.755770 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7e90d28e-739e-4b37-acdb-6997186a73e8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rzskr\" (UID: \"7e90d28e-739e-4b37-acdb-6997186a73e8\") " pod="openshift-insights/insights-runtime-extractor-rzskr" Jul 27 17:28:43.755992 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:43.755826 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7e90d28e-739e-4b37-acdb-6997186a73e8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rzskr\" (UID: \"7e90d28e-739e-4b37-acdb-6997186a73e8\") " pod="openshift-insights/insights-runtime-extractor-rzskr" Jul 27 17:28:43.755992 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:43.755910 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/91b9ae5b-5c22-41ab-8845-cd9f65146cbc-metrics-tls\") pod \"dns-default-dpkfg\" (UID: \"91b9ae5b-5c22-41ab-8845-cd9f65146cbc\") " pod="openshift-dns/dns-default-dpkfg" Jul 27 17:28:43.755992 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:43.755928 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 17:28:43.755992 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:43.755992 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e90d28e-739e-4b37-acdb-6997186a73e8-insights-runtime-extractor-tls podName:7e90d28e-739e-4b37-acdb-6997186a73e8 nodeName:}" failed. No retries permitted until 2026-07-27 17:28:45.755971716 +0000 UTC m=+43.197186742 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7e90d28e-739e-4b37-acdb-6997186a73e8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rzskr" (UID: "7e90d28e-739e-4b37-acdb-6997186a73e8") : secret "insights-runtime-extractor-tls" not found Jul 27 17:28:43.756196 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:43.756039 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 17:28:43.756196 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:43.756084 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91b9ae5b-5c22-41ab-8845-cd9f65146cbc-metrics-tls podName:91b9ae5b-5c22-41ab-8845-cd9f65146cbc nodeName:}" failed. No retries permitted until 2026-07-27 17:28:45.756070871 +0000 UTC m=+43.197285839 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/91b9ae5b-5c22-41ab-8845-cd9f65146cbc-metrics-tls") pod "dns-default-dpkfg" (UID: "91b9ae5b-5c22-41ab-8845-cd9f65146cbc") : secret "dns-default-metrics-tls" not found Jul 27 17:28:43.756196 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:43.756160 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 27 17:28:43.756317 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:43.756230 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7e90d28e-739e-4b37-acdb-6997186a73e8-kube-rbac-proxy-cm podName:7e90d28e-739e-4b37-acdb-6997186a73e8 nodeName:}" failed. No retries permitted until 2026-07-27 17:28:45.756213345 +0000 UTC m=+43.197428332 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7e90d28e-739e-4b37-acdb-6997186a73e8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rzskr" (UID: "7e90d28e-739e-4b37-acdb-6997186a73e8") : configmap "kube-rbac-proxy" not found Jul 27 17:28:45.338012 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:45.337975 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gc5j9" event={"ID":"1596f129-a503-4513-bb1f-3b7ac6339c51","Type":"ContainerStarted","Data":"2bb96812b8f7bda7c92b7973d12d4ec2b8b14855513a64a7cc14557b30741a89"} Jul 27 17:28:45.773348 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:45.773301 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7e90d28e-739e-4b37-acdb-6997186a73e8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rzskr\" (UID: \"7e90d28e-739e-4b37-acdb-6997186a73e8\") " pod="openshift-insights/insights-runtime-extractor-rzskr" Jul 27 17:28:45.773537 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:45.773356 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7e90d28e-739e-4b37-acdb-6997186a73e8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rzskr\" (UID: \"7e90d28e-739e-4b37-acdb-6997186a73e8\") " pod="openshift-insights/insights-runtime-extractor-rzskr" Jul 27 17:28:45.773537 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:45.773397 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/91b9ae5b-5c22-41ab-8845-cd9f65146cbc-metrics-tls\") pod \"dns-default-dpkfg\" (UID: \"91b9ae5b-5c22-41ab-8845-cd9f65146cbc\") " pod="openshift-dns/dns-default-dpkfg" Jul 27 17:28:45.773537 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:45.773436 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 27 17:28:45.773537 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:45.773504 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7e90d28e-739e-4b37-acdb-6997186a73e8-kube-rbac-proxy-cm podName:7e90d28e-739e-4b37-acdb-6997186a73e8 nodeName:}" failed. No retries permitted until 2026-07-27 17:28:49.773484946 +0000 UTC m=+47.214699909 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7e90d28e-739e-4b37-acdb-6997186a73e8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rzskr" (UID: "7e90d28e-739e-4b37-acdb-6997186a73e8") : configmap "kube-rbac-proxy" not found Jul 27 17:28:45.773694 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:45.773550 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 17:28:45.773694 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:45.773569 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 17:28:45.773694 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:45.773601 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91b9ae5b-5c22-41ab-8845-cd9f65146cbc-metrics-tls podName:91b9ae5b-5c22-41ab-8845-cd9f65146cbc nodeName:}" failed. No retries permitted until 2026-07-27 17:28:49.773584075 +0000 UTC m=+47.214799027 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/91b9ae5b-5c22-41ab-8845-cd9f65146cbc-metrics-tls") pod "dns-default-dpkfg" (UID: "91b9ae5b-5c22-41ab-8845-cd9f65146cbc") : secret "dns-default-metrics-tls" not found Jul 27 17:28:45.773694 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:45.773629 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e90d28e-739e-4b37-acdb-6997186a73e8-insights-runtime-extractor-tls podName:7e90d28e-739e-4b37-acdb-6997186a73e8 nodeName:}" failed. No retries permitted until 2026-07-27 17:28:49.773615817 +0000 UTC m=+47.214830768 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7e90d28e-739e-4b37-acdb-6997186a73e8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rzskr" (UID: "7e90d28e-739e-4b37-acdb-6997186a73e8") : secret "insights-runtime-extractor-tls" not found Jul 27 17:28:49.804423 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:49.804384 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7e90d28e-739e-4b37-acdb-6997186a73e8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rzskr\" (UID: \"7e90d28e-739e-4b37-acdb-6997186a73e8\") " pod="openshift-insights/insights-runtime-extractor-rzskr" Jul 27 17:28:49.804807 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:49.804440 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/91b9ae5b-5c22-41ab-8845-cd9f65146cbc-metrics-tls\") pod \"dns-default-dpkfg\" (UID: \"91b9ae5b-5c22-41ab-8845-cd9f65146cbc\") " pod="openshift-dns/dns-default-dpkfg" Jul 27 17:28:49.804807 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:49.804495 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7e90d28e-739e-4b37-acdb-6997186a73e8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rzskr\" (UID: \"7e90d28e-739e-4b37-acdb-6997186a73e8\") " pod="openshift-insights/insights-runtime-extractor-rzskr" Jul 27 17:28:49.804807 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:49.804557 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 17:28:49.804807 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:49.804579 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 27 17:28:49.804807 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:49.804637 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e90d28e-739e-4b37-acdb-6997186a73e8-insights-runtime-extractor-tls podName:7e90d28e-739e-4b37-acdb-6997186a73e8 nodeName:}" failed. No retries permitted until 2026-07-27 17:28:57.804616786 +0000 UTC m=+55.245831738 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7e90d28e-739e-4b37-acdb-6997186a73e8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rzskr" (UID: "7e90d28e-739e-4b37-acdb-6997186a73e8") : secret "insights-runtime-extractor-tls" not found Jul 27 17:28:49.804807 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:28:49.804657 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7e90d28e-739e-4b37-acdb-6997186a73e8-kube-rbac-proxy-cm podName:7e90d28e-739e-4b37-acdb-6997186a73e8 nodeName:}" failed. No retries permitted until 2026-07-27 17:28:57.804649107 +0000 UTC m=+55.245864063 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7e90d28e-739e-4b37-acdb-6997186a73e8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rzskr" (UID: "7e90d28e-739e-4b37-acdb-6997186a73e8") : configmap "kube-rbac-proxy" not found Jul 27 17:28:49.807911 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:49.807886 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/91b9ae5b-5c22-41ab-8845-cd9f65146cbc-metrics-tls\") pod \"dns-default-dpkfg\" (UID: \"91b9ae5b-5c22-41ab-8845-cd9f65146cbc\") " pod="openshift-dns/dns-default-dpkfg" Jul 27 17:28:50.087690 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:50.087599 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dpkfg" Jul 27 17:28:50.219826 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:50.219560 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-gc5j9" podStartSLOduration=6.621257986 podStartE2EDuration="9.219539366s" totalImagesPullingTime="2.59828138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:28:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:28:45.36904701 +0000 UTC m=+42.810261984" watchObservedRunningTime="2026-07-27 17:28:50.219539366 +0000 UTC m=+47.660754338" Jul 27 17:28:50.220360 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:50.220334 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dpkfg"] Jul 27 17:28:50.225392 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:50.225371 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91b9ae5b_5c22_41ab_8845_cd9f65146cbc.slice/crio-f4fb28120084831c6b8086c4d03a83c5d0ac74bc2e53a7199c93569c33ae41ee WatchSource:0}: Error finding container f4fb28120084831c6b8086c4d03a83c5d0ac74bc2e53a7199c93569c33ae41ee: Status 404 returned error can't find the container with id f4fb28120084831c6b8086c4d03a83c5d0ac74bc2e53a7199c93569c33ae41ee Jul 27 17:28:50.350797 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:50.350706 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dpkfg" event={"ID":"91b9ae5b-5c22-41ab-8845-cd9f65146cbc","Type":"ContainerStarted","Data":"f4fb28120084831c6b8086c4d03a83c5d0ac74bc2e53a7199c93569c33ae41ee"} Jul 27 17:28:53.359324 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:53.359220 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dpkfg" event={"ID":"91b9ae5b-5c22-41ab-8845-cd9f65146cbc","Type":"ContainerStarted","Data":"ca313f6302f7232ed1f06c256ff28dd9cc50625fdfc990d57a06f6f7b0f00262"} Jul 27 17:28:53.359324 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:53.359283 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dpkfg" event={"ID":"91b9ae5b-5c22-41ab-8845-cd9f65146cbc","Type":"ContainerStarted","Data":"a3a8a70cb05b2079489a3cb10c9e202f23ed531d471d678fcf6f78895037ea46"} Jul 27 17:28:53.359324 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:53.359322 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-dpkfg" Jul 27 17:28:53.377439 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:53.377390 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-dpkfg" podStartSLOduration=9.541168344 podStartE2EDuration="12.377376739s" totalImagesPullingTime="2.836208395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:28:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:28:53.377024441 +0000 UTC m=+50.818239414" watchObservedRunningTime="2026-07-27 17:28:53.377376739 +0000 UTC m=+50.818591713" Jul 27 17:28:57.864265 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:57.864222 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7e90d28e-739e-4b37-acdb-6997186a73e8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rzskr\" (UID: \"7e90d28e-739e-4b37-acdb-6997186a73e8\") " pod="openshift-insights/insights-runtime-extractor-rzskr" Jul 27 17:28:57.864855 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:57.864293 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7e90d28e-739e-4b37-acdb-6997186a73e8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rzskr\" (UID: \"7e90d28e-739e-4b37-acdb-6997186a73e8\") " pod="openshift-insights/insights-runtime-extractor-rzskr" Jul 27 17:28:57.864855 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:57.864785 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7e90d28e-739e-4b37-acdb-6997186a73e8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rzskr\" (UID: \"7e90d28e-739e-4b37-acdb-6997186a73e8\") " pod="openshift-insights/insights-runtime-extractor-rzskr" Jul 27 17:28:57.867334 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:57.867311 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7e90d28e-739e-4b37-acdb-6997186a73e8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rzskr\" (UID: \"7e90d28e-739e-4b37-acdb-6997186a73e8\") " pod="openshift-insights/insights-runtime-extractor-rzskr" Jul 27 17:28:57.946308 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:57.946275 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rzskr" Jul 27 17:28:58.067508 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:58.067425 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rzskr"] Jul 27 17:28:58.070128 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:28:58.070105 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e90d28e_739e_4b37_acdb_6997186a73e8.slice/crio-c39c9584fb0dc7c053e5ef183128f021c5aa9f65ea10eb052b2054da8438c2d8 WatchSource:0}: Error finding container c39c9584fb0dc7c053e5ef183128f021c5aa9f65ea10eb052b2054da8438c2d8: Status 404 returned error can't find the container with id c39c9584fb0dc7c053e5ef183128f021c5aa9f65ea10eb052b2054da8438c2d8 Jul 27 17:28:58.371557 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:28:58.371510 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rzskr" event={"ID":"7e90d28e-739e-4b37-acdb-6997186a73e8","Type":"ContainerStarted","Data":"c39c9584fb0dc7c053e5ef183128f021c5aa9f65ea10eb052b2054da8438c2d8"} Jul 27 17:29:03.364436 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:03.364403 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-dpkfg" Jul 27 17:29:04.389609 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:04.389507 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rzskr" event={"ID":"7e90d28e-739e-4b37-acdb-6997186a73e8","Type":"ContainerStarted","Data":"76b6fbc678c8e6652e63714fb35357e39f9143b9a8f03cd413b9185f93df96d1"} Jul 27 17:29:04.552237 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:04.552204 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-g9grw"] Jul 27 17:29:04.554771 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:04.554742 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-g9grw" Jul 27 17:29:04.561824 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:04.561755 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-9rt4h\"" Jul 27 17:29:04.561997 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:04.561847 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 27 17:29:04.562384 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:04.562368 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 27 17:29:04.578608 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:04.578575 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-g9grw"] Jul 27 17:29:04.605858 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:04.605817 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f60c07b8-db05-40f2-b0ac-64df9e16e9e9-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-g9grw\" (UID: \"f60c07b8-db05-40f2-b0ac-64df9e16e9e9\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-g9grw" Jul 27 17:29:04.606041 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:04.605868 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f60c07b8-db05-40f2-b0ac-64df9e16e9e9-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-g9grw\" (UID: \"f60c07b8-db05-40f2-b0ac-64df9e16e9e9\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-g9grw" Jul 27 17:29:04.706311 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:04.706226 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f60c07b8-db05-40f2-b0ac-64df9e16e9e9-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-g9grw\" (UID: \"f60c07b8-db05-40f2-b0ac-64df9e16e9e9\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-g9grw" Jul 27 17:29:04.706311 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:04.706270 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f60c07b8-db05-40f2-b0ac-64df9e16e9e9-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-g9grw\" (UID: \"f60c07b8-db05-40f2-b0ac-64df9e16e9e9\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-g9grw" Jul 27 17:29:04.706869 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:04.706843 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f60c07b8-db05-40f2-b0ac-64df9e16e9e9-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-g9grw\" (UID: \"f60c07b8-db05-40f2-b0ac-64df9e16e9e9\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-g9grw" Jul 27 17:29:04.707819 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:04.707796 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-bgt4v"] Jul 27 17:29:04.708725 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:04.708708 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f60c07b8-db05-40f2-b0ac-64df9e16e9e9-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-g9grw\" (UID: \"f60c07b8-db05-40f2-b0ac-64df9e16e9e9\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-g9grw" Jul 27 17:29:04.709695 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:04.709678 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-bgt4v" Jul 27 17:29:04.715773 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:04.715753 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 27 17:29:04.715877 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:04.715752 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 27 17:29:04.715877 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:04.715809 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-zrk9l\"" Jul 27 17:29:04.745238 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:04.745205 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-bgt4v"] Jul 27 17:29:04.806659 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:04.806626 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5vnf\" (UniqueName: \"kubernetes.io/projected/ad4401c9-a50e-46d6-a1f6-e4d8bf591064-kube-api-access-w5vnf\") pod \"downloads-7c97dc4555-bgt4v\" (UID: \"ad4401c9-a50e-46d6-a1f6-e4d8bf591064\") " pod="openshift-console/downloads-7c97dc4555-bgt4v" Jul 27 17:29:04.865801 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:04.865762 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-g9grw" Jul 27 17:29:04.870682 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:04.870656 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6bccbd7d69-hmc99"] Jul 27 17:29:04.872855 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:04.872833 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bccbd7d69-hmc99" Jul 27 17:29:04.875655 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:04.875638 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 27 17:29:04.876472 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:04.876452 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-lcj8p\"" Jul 27 17:29:04.876714 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:04.876702 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 27 17:29:04.884454 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:04.884425 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 27 17:29:04.889105 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:04.889080 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 27 17:29:04.900307 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:04.900275 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6bccbd7d69-hmc99"] Jul 27 17:29:04.907425 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:04.907398 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cf05ff2e-7a6f-4f26-aedc-302f49afdd34-ca-trust-extracted\") pod \"image-registry-6bccbd7d69-hmc99\" (UID: \"cf05ff2e-7a6f-4f26-aedc-302f49afdd34\") " pod="openshift-image-registry/image-registry-6bccbd7d69-hmc99" Jul 27 17:29:04.907577 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:04.907433 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cf05ff2e-7a6f-4f26-aedc-302f49afdd34-bound-sa-token\") pod \"image-registry-6bccbd7d69-hmc99\" (UID: \"cf05ff2e-7a6f-4f26-aedc-302f49afdd34\") " pod="openshift-image-registry/image-registry-6bccbd7d69-hmc99" Jul 27 17:29:04.907577 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:04.907460 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cf05ff2e-7a6f-4f26-aedc-302f49afdd34-registry-tls\") pod \"image-registry-6bccbd7d69-hmc99\" (UID: \"cf05ff2e-7a6f-4f26-aedc-302f49afdd34\") " pod="openshift-image-registry/image-registry-6bccbd7d69-hmc99" Jul 27 17:29:04.907577 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:04.907560 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cf05ff2e-7a6f-4f26-aedc-302f49afdd34-trusted-ca\") pod \"image-registry-6bccbd7d69-hmc99\" (UID: \"cf05ff2e-7a6f-4f26-aedc-302f49afdd34\") " pod="openshift-image-registry/image-registry-6bccbd7d69-hmc99" Jul 27 17:29:04.907693 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:04.907586 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cf05ff2e-7a6f-4f26-aedc-302f49afdd34-installation-pull-secrets\") pod \"image-registry-6bccbd7d69-hmc99\" (UID: \"cf05ff2e-7a6f-4f26-aedc-302f49afdd34\") " pod="openshift-image-registry/image-registry-6bccbd7d69-hmc99" Jul 27 17:29:04.907693 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:04.907640 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5vnf\" (UniqueName: \"kubernetes.io/projected/ad4401c9-a50e-46d6-a1f6-e4d8bf591064-kube-api-access-w5vnf\") pod \"downloads-7c97dc4555-bgt4v\" (UID: \"ad4401c9-a50e-46d6-a1f6-e4d8bf591064\") " pod="openshift-console/downloads-7c97dc4555-bgt4v" Jul 27 17:29:04.907693 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:04.907659 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cf05ff2e-7a6f-4f26-aedc-302f49afdd34-image-registry-private-configuration\") pod \"image-registry-6bccbd7d69-hmc99\" (UID: \"cf05ff2e-7a6f-4f26-aedc-302f49afdd34\") " pod="openshift-image-registry/image-registry-6bccbd7d69-hmc99" Jul 27 17:29:04.907693 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:04.907675 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldfzm\" (UniqueName: \"kubernetes.io/projected/cf05ff2e-7a6f-4f26-aedc-302f49afdd34-kube-api-access-ldfzm\") pod \"image-registry-6bccbd7d69-hmc99\" (UID: \"cf05ff2e-7a6f-4f26-aedc-302f49afdd34\") " pod="openshift-image-registry/image-registry-6bccbd7d69-hmc99" Jul 27 17:29:04.907979 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:04.907893 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cf05ff2e-7a6f-4f26-aedc-302f49afdd34-registry-certificates\") pod \"image-registry-6bccbd7d69-hmc99\" (UID: \"cf05ff2e-7a6f-4f26-aedc-302f49afdd34\") " pod="openshift-image-registry/image-registry-6bccbd7d69-hmc99" Jul 27 17:29:04.919061 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:04.919029 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5vnf\" (UniqueName: \"kubernetes.io/projected/ad4401c9-a50e-46d6-a1f6-e4d8bf591064-kube-api-access-w5vnf\") pod \"downloads-7c97dc4555-bgt4v\" (UID: \"ad4401c9-a50e-46d6-a1f6-e4d8bf591064\") " pod="openshift-console/downloads-7c97dc4555-bgt4v" Jul 27 17:29:05.008795 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:05.008762 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cf05ff2e-7a6f-4f26-aedc-302f49afdd34-registry-certificates\") pod \"image-registry-6bccbd7d69-hmc99\" (UID: \"cf05ff2e-7a6f-4f26-aedc-302f49afdd34\") " pod="openshift-image-registry/image-registry-6bccbd7d69-hmc99" Jul 27 17:29:05.008968 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:05.008821 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cf05ff2e-7a6f-4f26-aedc-302f49afdd34-ca-trust-extracted\") pod \"image-registry-6bccbd7d69-hmc99\" (UID: \"cf05ff2e-7a6f-4f26-aedc-302f49afdd34\") " pod="openshift-image-registry/image-registry-6bccbd7d69-hmc99" Jul 27 17:29:05.008968 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:05.008844 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cf05ff2e-7a6f-4f26-aedc-302f49afdd34-bound-sa-token\") pod \"image-registry-6bccbd7d69-hmc99\" (UID: \"cf05ff2e-7a6f-4f26-aedc-302f49afdd34\") " pod="openshift-image-registry/image-registry-6bccbd7d69-hmc99" Jul 27 17:29:05.008968 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:05.008869 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cf05ff2e-7a6f-4f26-aedc-302f49afdd34-registry-tls\") pod \"image-registry-6bccbd7d69-hmc99\" (UID: \"cf05ff2e-7a6f-4f26-aedc-302f49afdd34\") " pod="openshift-image-registry/image-registry-6bccbd7d69-hmc99" Jul 27 17:29:05.008968 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:05.008914 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cf05ff2e-7a6f-4f26-aedc-302f49afdd34-trusted-ca\") pod \"image-registry-6bccbd7d69-hmc99\" (UID: \"cf05ff2e-7a6f-4f26-aedc-302f49afdd34\") " pod="openshift-image-registry/image-registry-6bccbd7d69-hmc99" Jul 27 17:29:05.008968 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:05.008936 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cf05ff2e-7a6f-4f26-aedc-302f49afdd34-installation-pull-secrets\") pod \"image-registry-6bccbd7d69-hmc99\" (UID: \"cf05ff2e-7a6f-4f26-aedc-302f49afdd34\") " pod="openshift-image-registry/image-registry-6bccbd7d69-hmc99" Jul 27 17:29:05.009215 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:05.008976 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cf05ff2e-7a6f-4f26-aedc-302f49afdd34-image-registry-private-configuration\") pod \"image-registry-6bccbd7d69-hmc99\" (UID: \"cf05ff2e-7a6f-4f26-aedc-302f49afdd34\") " pod="openshift-image-registry/image-registry-6bccbd7d69-hmc99" Jul 27 17:29:05.009215 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:05.009000 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldfzm\" (UniqueName: \"kubernetes.io/projected/cf05ff2e-7a6f-4f26-aedc-302f49afdd34-kube-api-access-ldfzm\") pod \"image-registry-6bccbd7d69-hmc99\" (UID: \"cf05ff2e-7a6f-4f26-aedc-302f49afdd34\") " pod="openshift-image-registry/image-registry-6bccbd7d69-hmc99" Jul 27 17:29:05.009431 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:05.009398 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cf05ff2e-7a6f-4f26-aedc-302f49afdd34-ca-trust-extracted\") pod \"image-registry-6bccbd7d69-hmc99\" (UID: \"cf05ff2e-7a6f-4f26-aedc-302f49afdd34\") " pod="openshift-image-registry/image-registry-6bccbd7d69-hmc99" Jul 27 17:29:05.009766 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:05.009729 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cf05ff2e-7a6f-4f26-aedc-302f49afdd34-registry-certificates\") pod \"image-registry-6bccbd7d69-hmc99\" (UID: \"cf05ff2e-7a6f-4f26-aedc-302f49afdd34\") " pod="openshift-image-registry/image-registry-6bccbd7d69-hmc99" Jul 27 17:29:05.010075 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:05.010016 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cf05ff2e-7a6f-4f26-aedc-302f49afdd34-trusted-ca\") pod \"image-registry-6bccbd7d69-hmc99\" (UID: \"cf05ff2e-7a6f-4f26-aedc-302f49afdd34\") " pod="openshift-image-registry/image-registry-6bccbd7d69-hmc99" Jul 27 17:29:05.011100 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:05.011082 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-g9grw"] Jul 27 17:29:05.011758 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:05.011702 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cf05ff2e-7a6f-4f26-aedc-302f49afdd34-image-registry-private-configuration\") pod \"image-registry-6bccbd7d69-hmc99\" (UID: \"cf05ff2e-7a6f-4f26-aedc-302f49afdd34\") " pod="openshift-image-registry/image-registry-6bccbd7d69-hmc99" Jul 27 17:29:05.011988 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:05.011966 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cf05ff2e-7a6f-4f26-aedc-302f49afdd34-installation-pull-secrets\") pod \"image-registry-6bccbd7d69-hmc99\" (UID: \"cf05ff2e-7a6f-4f26-aedc-302f49afdd34\") " pod="openshift-image-registry/image-registry-6bccbd7d69-hmc99" Jul 27 17:29:05.012024 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:05.011992 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cf05ff2e-7a6f-4f26-aedc-302f49afdd34-registry-tls\") pod \"image-registry-6bccbd7d69-hmc99\" (UID: \"cf05ff2e-7a6f-4f26-aedc-302f49afdd34\") " pod="openshift-image-registry/image-registry-6bccbd7d69-hmc99" Jul 27 17:29:05.014085 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:29:05.014066 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf60c07b8_db05_40f2_b0ac_64df9e16e9e9.slice/crio-9b03a0b37ef6a7dcbe7a4d38f7ba27b31e3645b9a4226618090f8ea25b6e4060 WatchSource:0}: Error finding container 9b03a0b37ef6a7dcbe7a4d38f7ba27b31e3645b9a4226618090f8ea25b6e4060: Status 404 returned error can't find the container with id 9b03a0b37ef6a7dcbe7a4d38f7ba27b31e3645b9a4226618090f8ea25b6e4060 Jul 27 17:29:05.019957 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:05.019931 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cf05ff2e-7a6f-4f26-aedc-302f49afdd34-bound-sa-token\") pod \"image-registry-6bccbd7d69-hmc99\" (UID: \"cf05ff2e-7a6f-4f26-aedc-302f49afdd34\") " pod="openshift-image-registry/image-registry-6bccbd7d69-hmc99" Jul 27 17:29:05.020563 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:05.020545 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldfzm\" (UniqueName: \"kubernetes.io/projected/cf05ff2e-7a6f-4f26-aedc-302f49afdd34-kube-api-access-ldfzm\") pod \"image-registry-6bccbd7d69-hmc99\" (UID: \"cf05ff2e-7a6f-4f26-aedc-302f49afdd34\") " pod="openshift-image-registry/image-registry-6bccbd7d69-hmc99" Jul 27 17:29:05.020756 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:05.020743 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-bgt4v" Jul 27 17:29:05.141281 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:05.141253 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-bgt4v"] Jul 27 17:29:05.144077 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:29:05.144052 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad4401c9_a50e_46d6_a1f6_e4d8bf591064.slice/crio-08d06f96ef6fb0a51e44b5707b9d1ef230a4b74ac4d64138b816a8f78634c199 WatchSource:0}: Error finding container 08d06f96ef6fb0a51e44b5707b9d1ef230a4b74ac4d64138b816a8f78634c199: Status 404 returned error can't find the container with id 08d06f96ef6fb0a51e44b5707b9d1ef230a4b74ac4d64138b816a8f78634c199 Jul 27 17:29:05.189141 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:05.189110 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bccbd7d69-hmc99" Jul 27 17:29:05.314668 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:05.314603 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6bccbd7d69-hmc99"] Jul 27 17:29:05.318119 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:29:05.318030 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf05ff2e_7a6f_4f26_aedc_302f49afdd34.slice/crio-1a2c50fc81281c026b259d6ca90f413db7d141ae54a1bf4de115c45e7f3c20e0 WatchSource:0}: Error finding container 1a2c50fc81281c026b259d6ca90f413db7d141ae54a1bf4de115c45e7f3c20e0: Status 404 returned error can't find the container with id 1a2c50fc81281c026b259d6ca90f413db7d141ae54a1bf4de115c45e7f3c20e0 Jul 27 17:29:05.395507 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:05.395363 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6bccbd7d69-hmc99" event={"ID":"cf05ff2e-7a6f-4f26-aedc-302f49afdd34","Type":"ContainerStarted","Data":"1a2c50fc81281c026b259d6ca90f413db7d141ae54a1bf4de115c45e7f3c20e0"} Jul 27 17:29:05.397309 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:05.397230 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-bgt4v" event={"ID":"ad4401c9-a50e-46d6-a1f6-e4d8bf591064","Type":"ContainerStarted","Data":"08d06f96ef6fb0a51e44b5707b9d1ef230a4b74ac4d64138b816a8f78634c199"} Jul 27 17:29:05.399682 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:05.399636 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-g9grw" event={"ID":"f60c07b8-db05-40f2-b0ac-64df9e16e9e9","Type":"ContainerStarted","Data":"9b03a0b37ef6a7dcbe7a4d38f7ba27b31e3645b9a4226618090f8ea25b6e4060"} Jul 27 17:29:06.406283 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:06.406147 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-g9grw" event={"ID":"f60c07b8-db05-40f2-b0ac-64df9e16e9e9","Type":"ContainerStarted","Data":"293304e86811d0183cc3a26efee09a86fcf25ea30f2fae72cae0ec851485ced8"} Jul 27 17:29:06.408322 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:06.408289 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6bccbd7d69-hmc99" event={"ID":"cf05ff2e-7a6f-4f26-aedc-302f49afdd34","Type":"ContainerStarted","Data":"39348ead5f36df244ba12ca6d15a18f6489ea4ea08da929730791301ebec4173"} Jul 27 17:29:06.408452 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:06.408436 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6bccbd7d69-hmc99" Jul 27 17:29:06.421214 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:06.421164 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-g9grw" podStartSLOduration=1.297262834 podStartE2EDuration="2.421145612s" totalImagesPullingTime="1.123882778s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:29:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:29:06.420993752 +0000 UTC m=+63.862208726" watchObservedRunningTime="2026-07-27 17:29:06.421145612 +0000 UTC m=+63.862360586" Jul 27 17:29:06.444726 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:06.444364 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6bccbd7d69-hmc99" podStartSLOduration=2.4443425579999998 podStartE2EDuration="2.444342558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:29:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:29:06.442703356 +0000 UTC m=+63.883918330" watchObservedRunningTime="2026-07-27 17:29:06.444342558 +0000 UTC m=+63.885557585" Jul 27 17:29:07.318884 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:07.318853 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nh6qf" Jul 27 17:29:07.730693 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:07.730601 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7209ee7f-3019-4465-8175-3317741f9b09-original-pull-secret\") pod \"global-pull-secret-syncer-d4nqw\" (UID: \"7209ee7f-3019-4465-8175-3317741f9b09\") " pod="kube-system/global-pull-secret-syncer-d4nqw" Jul 27 17:29:07.732707 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:07.732680 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 27 17:29:07.745562 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:07.744444 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7209ee7f-3019-4465-8175-3317741f9b09-original-pull-secret\") pod \"global-pull-secret-syncer-d4nqw\" (UID: \"7209ee7f-3019-4465-8175-3317741f9b09\") " pod="kube-system/global-pull-secret-syncer-d4nqw" Jul 27 17:29:07.831590 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:07.831550 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5115219e-8201-45d1-8a3f-6db85d5e741d-metrics-certs\") pod \"network-metrics-daemon-69zjf\" (UID: \"5115219e-8201-45d1-8a3f-6db85d5e741d\") " pod="openshift-multus/network-metrics-daemon-69zjf" Jul 27 17:29:07.833816 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:07.833793 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 27 17:29:07.844553 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:07.844507 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5115219e-8201-45d1-8a3f-6db85d5e741d-metrics-certs\") pod \"network-metrics-daemon-69zjf\" (UID: \"5115219e-8201-45d1-8a3f-6db85d5e741d\") " pod="openshift-multus/network-metrics-daemon-69zjf" Jul 27 17:29:07.932573 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:07.932155 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p4xw\" (UniqueName: \"kubernetes.io/projected/ec0c6ea9-94a4-41b6-9e1f-cccf978294ab-kube-api-access-5p4xw\") pod \"network-check-target-jhzgt\" (UID: \"ec0c6ea9-94a4-41b6-9e1f-cccf978294ab\") " pod="openshift-network-diagnostics/network-check-target-jhzgt" Jul 27 17:29:07.934370 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:07.934343 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 27 17:29:07.935753 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:07.935728 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6sz2v\"" Jul 27 17:29:07.938484 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:07.938460 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d4nqw" Jul 27 17:29:07.944601 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:07.944578 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-69zjf" Jul 27 17:29:07.944785 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:07.944737 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 27 17:29:07.956070 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:07.956041 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p4xw\" (UniqueName: \"kubernetes.io/projected/ec0c6ea9-94a4-41b6-9e1f-cccf978294ab-kube-api-access-5p4xw\") pod \"network-check-target-jhzgt\" (UID: \"ec0c6ea9-94a4-41b6-9e1f-cccf978294ab\") " pod="openshift-network-diagnostics/network-check-target-jhzgt" Jul 27 17:29:08.039386 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:08.039352 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kcn4g\"" Jul 27 17:29:08.047346 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:08.047306 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhzgt" Jul 27 17:29:08.741845 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:08.741461 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jhzgt"] Jul 27 17:29:08.748080 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:29:08.747681 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec0c6ea9_94a4_41b6_9e1f_cccf978294ab.slice/crio-42530334457fd1f144bf49cdd2389284a7c787454ab30fc1bbdabf40c58f840c WatchSource:0}: Error finding container 42530334457fd1f144bf49cdd2389284a7c787454ab30fc1bbdabf40c58f840c: Status 404 returned error can't find the container with id 42530334457fd1f144bf49cdd2389284a7c787454ab30fc1bbdabf40c58f840c Jul 27 17:29:08.767090 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:08.767058 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-69zjf"] Jul 27 17:29:08.769819 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:29:08.769783 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5115219e_8201_45d1_8a3f_6db85d5e741d.slice/crio-12a49100b789014adafde33c929b7683b4f1babf0bcf5543cf0a4617fc84e529 WatchSource:0}: Error finding container 12a49100b789014adafde33c929b7683b4f1babf0bcf5543cf0a4617fc84e529: Status 404 returned error can't find the container with id 12a49100b789014adafde33c929b7683b4f1babf0bcf5543cf0a4617fc84e529 Jul 27 17:29:08.784948 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:08.784918 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-d4nqw"] Jul 27 17:29:08.789266 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:29:08.789187 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7209ee7f_3019_4465_8175_3317741f9b09.slice/crio-06b9785d45b202ba68777b11d25089249bbfb64891be1a14b8e042ef1f6692b7 WatchSource:0}: Error finding container 06b9785d45b202ba68777b11d25089249bbfb64891be1a14b8e042ef1f6692b7: Status 404 returned error can't find the container with id 06b9785d45b202ba68777b11d25089249bbfb64891be1a14b8e042ef1f6692b7 Jul 27 17:29:09.421998 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:09.421901 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-d4nqw" event={"ID":"7209ee7f-3019-4465-8175-3317741f9b09","Type":"ContainerStarted","Data":"06b9785d45b202ba68777b11d25089249bbfb64891be1a14b8e042ef1f6692b7"} Jul 27 17:29:09.424490 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:09.424406 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rzskr" event={"ID":"7e90d28e-739e-4b37-acdb-6997186a73e8","Type":"ContainerStarted","Data":"1dcce8a199dee4c3ea7933450f8f22976163e0ddfb4830c6216194e5d988e142"} Jul 27 17:29:09.426965 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:09.426938 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jhzgt" event={"ID":"ec0c6ea9-94a4-41b6-9e1f-cccf978294ab","Type":"ContainerStarted","Data":"42530334457fd1f144bf49cdd2389284a7c787454ab30fc1bbdabf40c58f840c"} Jul 27 17:29:09.429268 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:09.429199 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-69zjf" event={"ID":"5115219e-8201-45d1-8a3f-6db85d5e741d","Type":"ContainerStarted","Data":"12a49100b789014adafde33c929b7683b4f1babf0bcf5543cf0a4617fc84e529"} Jul 27 17:29:10.735783 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:10.735032 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-n974k"] Jul 27 17:29:10.756407 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:10.755650 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-n974k"] Jul 27 17:29:10.756407 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:10.755774 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-n974k" Jul 27 17:29:10.760253 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:10.760225 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-v5ltr\"" Jul 27 17:29:10.760499 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:10.760380 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 27 17:29:10.861053 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:10.861017 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c0b6a6c4-65f9-43b0-b176-aa722c1504df-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-n974k\" (UID: \"c0b6a6c4-65f9-43b0-b176-aa722c1504df\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-n974k" Jul 27 17:29:10.962061 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:10.962021 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c0b6a6c4-65f9-43b0-b176-aa722c1504df-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-n974k\" (UID: \"c0b6a6c4-65f9-43b0-b176-aa722c1504df\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-n974k" Jul 27 17:29:10.969204 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:10.968183 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c0b6a6c4-65f9-43b0-b176-aa722c1504df-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-n974k\" (UID: \"c0b6a6c4-65f9-43b0-b176-aa722c1504df\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-n974k" Jul 27 17:29:11.072357 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:11.072325 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-n974k" Jul 27 17:29:11.291648 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:11.290618 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-n974k"] Jul 27 17:29:11.296668 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:29:11.296613 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0b6a6c4_65f9_43b0_b176_aa722c1504df.slice/crio-5b2c42f15e22d03731c14d1005e75ec176af01630ac58bb1ff3336f19975fc78 WatchSource:0}: Error finding container 5b2c42f15e22d03731c14d1005e75ec176af01630ac58bb1ff3336f19975fc78: Status 404 returned error can't find the container with id 5b2c42f15e22d03731c14d1005e75ec176af01630ac58bb1ff3336f19975fc78 Jul 27 17:29:11.453502 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:11.453458 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-69zjf" event={"ID":"5115219e-8201-45d1-8a3f-6db85d5e741d","Type":"ContainerStarted","Data":"9ab84f94d7acdeff99ac7e6565f2a7a722ffe0ed6184e20f792bbdb78987726f"} Jul 27 17:29:11.453722 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:11.453593 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-69zjf" event={"ID":"5115219e-8201-45d1-8a3f-6db85d5e741d","Type":"ContainerStarted","Data":"154bf4e3d97329f04afce2d549423af128a8f40f0c11b70f99a18d76f2a88253"} Jul 27 17:29:11.456353 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:11.456328 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-n974k" event={"ID":"c0b6a6c4-65f9-43b0-b176-aa722c1504df","Type":"ContainerStarted","Data":"5b2c42f15e22d03731c14d1005e75ec176af01630ac58bb1ff3336f19975fc78"} Jul 27 17:29:11.468465 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:11.468252 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-69zjf" podStartSLOduration=66.238946823 podStartE2EDuration="1m8.468235772s" totalImagesPullingTime="2.229288949s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:28:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:29:11.468094576 +0000 UTC m=+68.909309551" watchObservedRunningTime="2026-07-27 17:29:11.468235772 +0000 UTC m=+68.909450748" Jul 27 17:29:13.974770 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:13.974728 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-675f4b885b-bggck"] Jul 27 17:29:13.976899 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:13.976868 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-675f4b885b-bggck" Jul 27 17:29:13.979185 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:13.979161 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 27 17:29:13.979281 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:13.979203 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 27 17:29:13.980136 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:13.979840 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 27 17:29:13.980136 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:13.979905 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 27 17:29:13.980136 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:13.979927 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 27 17:29:13.980136 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:13.979958 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-c76kr\"" Jul 27 17:29:13.989811 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:13.989791 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-675f4b885b-bggck"] Jul 27 17:29:14.086534 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:14.086467 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f1389cc5-47f4-4804-8c4e-836a5181bd76-console-oauth-config\") pod \"console-675f4b885b-bggck\" (UID: \"f1389cc5-47f4-4804-8c4e-836a5181bd76\") " pod="openshift-console/console-675f4b885b-bggck" Jul 27 17:29:14.086818 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:14.086562 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f1389cc5-47f4-4804-8c4e-836a5181bd76-oauth-serving-cert\") pod \"console-675f4b885b-bggck\" (UID: \"f1389cc5-47f4-4804-8c4e-836a5181bd76\") " pod="openshift-console/console-675f4b885b-bggck" Jul 27 17:29:14.086818 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:14.086593 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f1389cc5-47f4-4804-8c4e-836a5181bd76-console-config\") pod \"console-675f4b885b-bggck\" (UID: \"f1389cc5-47f4-4804-8c4e-836a5181bd76\") " pod="openshift-console/console-675f4b885b-bggck" Jul 27 17:29:14.086818 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:14.086620 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f1389cc5-47f4-4804-8c4e-836a5181bd76-console-serving-cert\") pod \"console-675f4b885b-bggck\" (UID: \"f1389cc5-47f4-4804-8c4e-836a5181bd76\") " pod="openshift-console/console-675f4b885b-bggck" Jul 27 17:29:14.086818 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:14.086650 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f1389cc5-47f4-4804-8c4e-836a5181bd76-service-ca\") pod \"console-675f4b885b-bggck\" (UID: \"f1389cc5-47f4-4804-8c4e-836a5181bd76\") " pod="openshift-console/console-675f4b885b-bggck" Jul 27 17:29:14.086818 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:14.086678 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42wcp\" (UniqueName: \"kubernetes.io/projected/f1389cc5-47f4-4804-8c4e-836a5181bd76-kube-api-access-42wcp\") pod \"console-675f4b885b-bggck\" (UID: \"f1389cc5-47f4-4804-8c4e-836a5181bd76\") " pod="openshift-console/console-675f4b885b-bggck" Jul 27 17:29:14.188103 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:14.188056 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f1389cc5-47f4-4804-8c4e-836a5181bd76-console-oauth-config\") pod \"console-675f4b885b-bggck\" (UID: \"f1389cc5-47f4-4804-8c4e-836a5181bd76\") " pod="openshift-console/console-675f4b885b-bggck" Jul 27 17:29:14.188277 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:14.188132 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f1389cc5-47f4-4804-8c4e-836a5181bd76-oauth-serving-cert\") pod \"console-675f4b885b-bggck\" (UID: \"f1389cc5-47f4-4804-8c4e-836a5181bd76\") " pod="openshift-console/console-675f4b885b-bggck" Jul 27 17:29:14.188277 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:14.188163 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f1389cc5-47f4-4804-8c4e-836a5181bd76-console-config\") pod \"console-675f4b885b-bggck\" (UID: \"f1389cc5-47f4-4804-8c4e-836a5181bd76\") " pod="openshift-console/console-675f4b885b-bggck" Jul 27 17:29:14.188277 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:14.188194 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f1389cc5-47f4-4804-8c4e-836a5181bd76-console-serving-cert\") pod \"console-675f4b885b-bggck\" (UID: \"f1389cc5-47f4-4804-8c4e-836a5181bd76\") " pod="openshift-console/console-675f4b885b-bggck" Jul 27 17:29:14.188440 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:14.188312 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f1389cc5-47f4-4804-8c4e-836a5181bd76-service-ca\") pod \"console-675f4b885b-bggck\" (UID: \"f1389cc5-47f4-4804-8c4e-836a5181bd76\") " pod="openshift-console/console-675f4b885b-bggck" Jul 27 17:29:14.188440 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:14.188362 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42wcp\" (UniqueName: \"kubernetes.io/projected/f1389cc5-47f4-4804-8c4e-836a5181bd76-kube-api-access-42wcp\") pod \"console-675f4b885b-bggck\" (UID: \"f1389cc5-47f4-4804-8c4e-836a5181bd76\") " pod="openshift-console/console-675f4b885b-bggck" Jul 27 17:29:14.189273 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:14.189211 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f1389cc5-47f4-4804-8c4e-836a5181bd76-console-config\") pod \"console-675f4b885b-bggck\" (UID: \"f1389cc5-47f4-4804-8c4e-836a5181bd76\") " pod="openshift-console/console-675f4b885b-bggck" Jul 27 17:29:14.189398 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:14.189288 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f1389cc5-47f4-4804-8c4e-836a5181bd76-service-ca\") pod \"console-675f4b885b-bggck\" (UID: \"f1389cc5-47f4-4804-8c4e-836a5181bd76\") " pod="openshift-console/console-675f4b885b-bggck" Jul 27 17:29:14.189398 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:14.189346 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f1389cc5-47f4-4804-8c4e-836a5181bd76-oauth-serving-cert\") pod \"console-675f4b885b-bggck\" (UID: \"f1389cc5-47f4-4804-8c4e-836a5181bd76\") " pod="openshift-console/console-675f4b885b-bggck" Jul 27 17:29:14.191185 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:14.191158 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f1389cc5-47f4-4804-8c4e-836a5181bd76-console-oauth-config\") pod \"console-675f4b885b-bggck\" (UID: \"f1389cc5-47f4-4804-8c4e-836a5181bd76\") " pod="openshift-console/console-675f4b885b-bggck" Jul 27 17:29:14.191293 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:14.191220 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f1389cc5-47f4-4804-8c4e-836a5181bd76-console-serving-cert\") pod \"console-675f4b885b-bggck\" (UID: \"f1389cc5-47f4-4804-8c4e-836a5181bd76\") " pod="openshift-console/console-675f4b885b-bggck" Jul 27 17:29:14.199716 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:14.199670 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42wcp\" (UniqueName: \"kubernetes.io/projected/f1389cc5-47f4-4804-8c4e-836a5181bd76-kube-api-access-42wcp\") pod \"console-675f4b885b-bggck\" (UID: \"f1389cc5-47f4-4804-8c4e-836a5181bd76\") " pod="openshift-console/console-675f4b885b-bggck" Jul 27 17:29:14.288794 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:14.288506 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-675f4b885b-bggck" Jul 27 17:29:17.192464 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:17.192383 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-675f4b885b-bggck"] Jul 27 17:29:17.196142 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:29:17.196113 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1389cc5_47f4_4804_8c4e_836a5181bd76.slice/crio-e1b9f25d0ad2e422007431b0062ad40ffca7b829ff63df36abe48ca786b81fc2 WatchSource:0}: Error finding container e1b9f25d0ad2e422007431b0062ad40ffca7b829ff63df36abe48ca786b81fc2: Status 404 returned error can't find the container with id e1b9f25d0ad2e422007431b0062ad40ffca7b829ff63df36abe48ca786b81fc2 Jul 27 17:29:17.481264 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:17.481185 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-n974k" event={"ID":"c0b6a6c4-65f9-43b0-b176-aa722c1504df","Type":"ContainerStarted","Data":"5bf30d4db0427257736db2ebc77c6a4eab42e248f5b8aa6a74c8a8b1b817f8db"} Jul 27 17:29:17.481538 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:17.481475 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-n974k" Jul 27 17:29:17.483263 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:17.483229 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-d4nqw" event={"ID":"7209ee7f-3019-4465-8175-3317741f9b09","Type":"ContainerStarted","Data":"dd78c13290915cc775411203ef5ce97de30563cf73f415f232f58723a74bd549"} Jul 27 17:29:17.487184 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:17.486977 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rzskr" event={"ID":"7e90d28e-739e-4b37-acdb-6997186a73e8","Type":"ContainerStarted","Data":"2d3b15abc5eee53cbe437aba2383134f95cb81e8bfe06860bf3db961c5fd2883"} Jul 27 17:29:17.487184 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:17.487133 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-n974k" Jul 27 17:29:17.488676 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:17.488655 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jhzgt" event={"ID":"ec0c6ea9-94a4-41b6-9e1f-cccf978294ab","Type":"ContainerStarted","Data":"ede5e2df62a19eb12dd9be1070f068c72a0de4f5c356426d18f226d01f69648e"} Jul 27 17:29:17.488769 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:17.488711 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jhzgt" Jul 27 17:29:17.490010 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:17.489989 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-675f4b885b-bggck" event={"ID":"f1389cc5-47f4-4804-8c4e-836a5181bd76","Type":"ContainerStarted","Data":"e1b9f25d0ad2e422007431b0062ad40ffca7b829ff63df36abe48ca786b81fc2"} Jul 27 17:29:17.496339 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:17.496298 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-n974k" podStartSLOduration=1.696793169 podStartE2EDuration="7.496282576s" totalImagesPullingTime="5.799489407s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:29:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:29:17.494855302 +0000 UTC m=+74.936070278" watchObservedRunningTime="2026-07-27 17:29:17.496282576 +0000 UTC m=+74.937497551" Jul 27 17:29:17.513610 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:17.513551 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-d4nqw" podStartSLOduration=66.269302192 podStartE2EDuration="1m14.513537746s" totalImagesPullingTime="8.244235554s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:28:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:29:17.513042133 +0000 UTC m=+74.954257111" watchObservedRunningTime="2026-07-27 17:29:17.513537746 +0000 UTC m=+74.954752717" Jul 27 17:29:17.527638 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:17.527395 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-jhzgt" podStartSLOduration=66.240538836 podStartE2EDuration="1m14.527378838s" totalImagesPullingTime="8.286840002s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:28:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:29:17.526756887 +0000 UTC m=+74.967971872" watchObservedRunningTime="2026-07-27 17:29:17.527378838 +0000 UTC m=+74.968593813" Jul 27 17:29:17.559246 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:17.559174 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-rzskr" podStartSLOduration=17.755419 podStartE2EDuration="36.559152532s" totalImagesPullingTime="18.803733532s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:28:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:29:17.558238935 +0000 UTC m=+74.999453913" watchObservedRunningTime="2026-07-27 17:29:17.559152532 +0000 UTC m=+75.000367706" Jul 27 17:29:23.138587 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.138557 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-cckjt"] Jul 27 17:29:23.170057 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.170026 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-cckjt"] Jul 27 17:29:23.170237 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.170070 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-wc86b"] Jul 27 17:29:23.170237 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.170164 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-cckjt" Jul 27 17:29:23.175146 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.175122 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-hcf9w\"" Jul 27 17:29:23.175604 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.175313 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 27 17:29:23.175604 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.175403 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 27 17:29:23.175604 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.175427 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 27 17:29:23.175604 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.175459 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 27 17:29:23.175604 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.175459 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 27 17:29:23.196889 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.196866 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-h67d4"] Jul 27 17:29:23.197020 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.197003 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wc86b" Jul 27 17:29:23.199343 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.199320 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 27 17:29:23.199468 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.199419 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-9jxf6\"" Jul 27 17:29:23.199720 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.199703 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 27 17:29:23.199804 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.199744 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 27 17:29:23.212098 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.212076 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-wc86b"] Jul 27 17:29:23.212221 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.212208 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-h67d4" Jul 27 17:29:23.214300 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.214281 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 27 17:29:23.214414 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.214340 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 27 17:29:23.214414 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.214289 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 27 17:29:23.214414 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.214289 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-fhlh7\"" Jul 27 17:29:23.262918 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.262887 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6269d5fd-1774-4aef-9cc7-2296db21f5a3-metrics-client-ca\") pod \"node-exporter-h67d4\" (UID: \"6269d5fd-1774-4aef-9cc7-2296db21f5a3\") " pod="openshift-monitoring/node-exporter-h67d4" Jul 27 17:29:23.262918 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.262926 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzzsv\" (UniqueName: \"kubernetes.io/projected/6269d5fd-1774-4aef-9cc7-2296db21f5a3-kube-api-access-tzzsv\") pod \"node-exporter-h67d4\" (UID: \"6269d5fd-1774-4aef-9cc7-2296db21f5a3\") " pod="openshift-monitoring/node-exporter-h67d4" Jul 27 17:29:23.263162 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.262945 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6269d5fd-1774-4aef-9cc7-2296db21f5a3-node-exporter-wtmp\") pod \"node-exporter-h67d4\" (UID: \"6269d5fd-1774-4aef-9cc7-2296db21f5a3\") " pod="openshift-monitoring/node-exporter-h67d4" Jul 27 17:29:23.263162 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.263008 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9e2569a9-2352-4e32-96b1-fc35089c290e-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-wc86b\" (UID: \"9e2569a9-2352-4e32-96b1-fc35089c290e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wc86b" Jul 27 17:29:23.263162 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.263056 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6269d5fd-1774-4aef-9cc7-2296db21f5a3-node-exporter-accelerators-collector-config\") pod \"node-exporter-h67d4\" (UID: \"6269d5fd-1774-4aef-9cc7-2296db21f5a3\") " pod="openshift-monitoring/node-exporter-h67d4" Jul 27 17:29:23.263162 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.263086 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6269d5fd-1774-4aef-9cc7-2296db21f5a3-sys\") pod \"node-exporter-h67d4\" (UID: \"6269d5fd-1774-4aef-9cc7-2296db21f5a3\") " pod="openshift-monitoring/node-exporter-h67d4" Jul 27 17:29:23.263162 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.263110 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lv7v\" (UniqueName: \"kubernetes.io/projected/a0aee75d-e83d-4d0e-a70e-edea165b58c7-kube-api-access-9lv7v\") pod \"openshift-state-metrics-b8f49f5c6-cckjt\" (UID: \"a0aee75d-e83d-4d0e-a70e-edea165b58c7\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-cckjt" Jul 27 17:29:23.263162 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.263128 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6269d5fd-1774-4aef-9cc7-2296db21f5a3-node-exporter-textfile\") pod \"node-exporter-h67d4\" (UID: \"6269d5fd-1774-4aef-9cc7-2296db21f5a3\") " pod="openshift-monitoring/node-exporter-h67d4" Jul 27 17:29:23.263435 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.263183 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e2569a9-2352-4e32-96b1-fc35089c290e-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-wc86b\" (UID: \"9e2569a9-2352-4e32-96b1-fc35089c290e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wc86b" Jul 27 17:29:23.263435 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.263222 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9e2569a9-2352-4e32-96b1-fc35089c290e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-wc86b\" (UID: \"9e2569a9-2352-4e32-96b1-fc35089c290e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wc86b" Jul 27 17:29:23.263435 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.263278 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh6nq\" (UniqueName: \"kubernetes.io/projected/9e2569a9-2352-4e32-96b1-fc35089c290e-kube-api-access-qh6nq\") pod \"kube-state-metrics-56d9f944b4-wc86b\" (UID: \"9e2569a9-2352-4e32-96b1-fc35089c290e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wc86b" Jul 27 17:29:23.263435 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.263357 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9e2569a9-2352-4e32-96b1-fc35089c290e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-wc86b\" (UID: \"9e2569a9-2352-4e32-96b1-fc35089c290e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wc86b" Jul 27 17:29:23.263435 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.263402 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6269d5fd-1774-4aef-9cc7-2296db21f5a3-root\") pod \"node-exporter-h67d4\" (UID: \"6269d5fd-1774-4aef-9cc7-2296db21f5a3\") " pod="openshift-monitoring/node-exporter-h67d4" Jul 27 17:29:23.263664 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.263460 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a0aee75d-e83d-4d0e-a70e-edea165b58c7-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-cckjt\" (UID: \"a0aee75d-e83d-4d0e-a70e-edea165b58c7\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-cckjt" Jul 27 17:29:23.263664 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.263486 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9e2569a9-2352-4e32-96b1-fc35089c290e-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-wc86b\" (UID: \"9e2569a9-2352-4e32-96b1-fc35089c290e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wc86b" Jul 27 17:29:23.263664 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.263541 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6269d5fd-1774-4aef-9cc7-2296db21f5a3-node-exporter-tls\") pod \"node-exporter-h67d4\" (UID: \"6269d5fd-1774-4aef-9cc7-2296db21f5a3\") " pod="openshift-monitoring/node-exporter-h67d4" Jul 27 17:29:23.263664 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.263565 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6269d5fd-1774-4aef-9cc7-2296db21f5a3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h67d4\" (UID: \"6269d5fd-1774-4aef-9cc7-2296db21f5a3\") " pod="openshift-monitoring/node-exporter-h67d4" Jul 27 17:29:23.263664 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.263604 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a0aee75d-e83d-4d0e-a70e-edea165b58c7-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-cckjt\" (UID: \"a0aee75d-e83d-4d0e-a70e-edea165b58c7\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-cckjt" Jul 27 17:29:23.263664 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.263633 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0aee75d-e83d-4d0e-a70e-edea165b58c7-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-cckjt\" (UID: \"a0aee75d-e83d-4d0e-a70e-edea165b58c7\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-cckjt" Jul 27 17:29:23.364825 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.364791 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9e2569a9-2352-4e32-96b1-fc35089c290e-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-wc86b\" (UID: \"9e2569a9-2352-4e32-96b1-fc35089c290e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wc86b" Jul 27 17:29:23.364825 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.364831 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6269d5fd-1774-4aef-9cc7-2296db21f5a3-node-exporter-accelerators-collector-config\") pod \"node-exporter-h67d4\" (UID: \"6269d5fd-1774-4aef-9cc7-2296db21f5a3\") " pod="openshift-monitoring/node-exporter-h67d4" Jul 27 17:29:23.365089 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.364854 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6269d5fd-1774-4aef-9cc7-2296db21f5a3-sys\") pod \"node-exporter-h67d4\" (UID: \"6269d5fd-1774-4aef-9cc7-2296db21f5a3\") " pod="openshift-monitoring/node-exporter-h67d4" Jul 27 17:29:23.365089 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.364884 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lv7v\" (UniqueName: \"kubernetes.io/projected/a0aee75d-e83d-4d0e-a70e-edea165b58c7-kube-api-access-9lv7v\") pod \"openshift-state-metrics-b8f49f5c6-cckjt\" (UID: \"a0aee75d-e83d-4d0e-a70e-edea165b58c7\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-cckjt" Jul 27 17:29:23.365089 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.364947 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6269d5fd-1774-4aef-9cc7-2296db21f5a3-sys\") pod \"node-exporter-h67d4\" (UID: \"6269d5fd-1774-4aef-9cc7-2296db21f5a3\") " pod="openshift-monitoring/node-exporter-h67d4" Jul 27 17:29:23.365089 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.365008 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6269d5fd-1774-4aef-9cc7-2296db21f5a3-node-exporter-textfile\") pod \"node-exporter-h67d4\" (UID: \"6269d5fd-1774-4aef-9cc7-2296db21f5a3\") " pod="openshift-monitoring/node-exporter-h67d4" Jul 27 17:29:23.365089 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.365065 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e2569a9-2352-4e32-96b1-fc35089c290e-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-wc86b\" (UID: \"9e2569a9-2352-4e32-96b1-fc35089c290e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wc86b" Jul 27 17:29:23.365316 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.365099 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9e2569a9-2352-4e32-96b1-fc35089c290e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-wc86b\" (UID: \"9e2569a9-2352-4e32-96b1-fc35089c290e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wc86b" Jul 27 17:29:23.365316 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.365128 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh6nq\" (UniqueName: \"kubernetes.io/projected/9e2569a9-2352-4e32-96b1-fc35089c290e-kube-api-access-qh6nq\") pod \"kube-state-metrics-56d9f944b4-wc86b\" (UID: \"9e2569a9-2352-4e32-96b1-fc35089c290e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wc86b" Jul 27 17:29:23.365316 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.365175 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9e2569a9-2352-4e32-96b1-fc35089c290e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-wc86b\" (UID: \"9e2569a9-2352-4e32-96b1-fc35089c290e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wc86b" Jul 27 17:29:23.365316 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.365211 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6269d5fd-1774-4aef-9cc7-2296db21f5a3-root\") pod \"node-exporter-h67d4\" (UID: \"6269d5fd-1774-4aef-9cc7-2296db21f5a3\") " pod="openshift-monitoring/node-exporter-h67d4" Jul 27 17:29:23.365316 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.365247 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a0aee75d-e83d-4d0e-a70e-edea165b58c7-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-cckjt\" (UID: \"a0aee75d-e83d-4d0e-a70e-edea165b58c7\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-cckjt" Jul 27 17:29:23.365316 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.365272 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9e2569a9-2352-4e32-96b1-fc35089c290e-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-wc86b\" (UID: \"9e2569a9-2352-4e32-96b1-fc35089c290e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wc86b" Jul 27 17:29:23.365316 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.365303 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6269d5fd-1774-4aef-9cc7-2296db21f5a3-node-exporter-tls\") pod \"node-exporter-h67d4\" (UID: \"6269d5fd-1774-4aef-9cc7-2296db21f5a3\") " pod="openshift-monitoring/node-exporter-h67d4" Jul 27 17:29:23.365675 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.365327 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6269d5fd-1774-4aef-9cc7-2296db21f5a3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h67d4\" (UID: \"6269d5fd-1774-4aef-9cc7-2296db21f5a3\") " pod="openshift-monitoring/node-exporter-h67d4" Jul 27 17:29:23.365675 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.365330 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6269d5fd-1774-4aef-9cc7-2296db21f5a3-node-exporter-textfile\") pod \"node-exporter-h67d4\" (UID: \"6269d5fd-1774-4aef-9cc7-2296db21f5a3\") " pod="openshift-monitoring/node-exporter-h67d4" Jul 27 17:29:23.365675 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.365360 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a0aee75d-e83d-4d0e-a70e-edea165b58c7-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-cckjt\" (UID: \"a0aee75d-e83d-4d0e-a70e-edea165b58c7\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-cckjt" Jul 27 17:29:23.365675 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.365386 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0aee75d-e83d-4d0e-a70e-edea165b58c7-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-cckjt\" (UID: \"a0aee75d-e83d-4d0e-a70e-edea165b58c7\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-cckjt" Jul 27 17:29:23.365675 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.365415 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6269d5fd-1774-4aef-9cc7-2296db21f5a3-metrics-client-ca\") pod \"node-exporter-h67d4\" (UID: \"6269d5fd-1774-4aef-9cc7-2296db21f5a3\") " pod="openshift-monitoring/node-exporter-h67d4" Jul 27 17:29:23.365675 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.365455 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzzsv\" (UniqueName: \"kubernetes.io/projected/6269d5fd-1774-4aef-9cc7-2296db21f5a3-kube-api-access-tzzsv\") pod \"node-exporter-h67d4\" (UID: \"6269d5fd-1774-4aef-9cc7-2296db21f5a3\") " pod="openshift-monitoring/node-exporter-h67d4" Jul 27 17:29:23.365675 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.365480 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6269d5fd-1774-4aef-9cc7-2296db21f5a3-node-exporter-wtmp\") pod \"node-exporter-h67d4\" (UID: \"6269d5fd-1774-4aef-9cc7-2296db21f5a3\") " pod="openshift-monitoring/node-exporter-h67d4" Jul 27 17:29:23.365675 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.365511 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6269d5fd-1774-4aef-9cc7-2296db21f5a3-node-exporter-accelerators-collector-config\") pod \"node-exporter-h67d4\" (UID: \"6269d5fd-1774-4aef-9cc7-2296db21f5a3\") " pod="openshift-monitoring/node-exporter-h67d4" Jul 27 17:29:23.365675 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.365660 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6269d5fd-1774-4aef-9cc7-2296db21f5a3-node-exporter-wtmp\") pod \"node-exporter-h67d4\" (UID: \"6269d5fd-1774-4aef-9cc7-2296db21f5a3\") " pod="openshift-monitoring/node-exporter-h67d4" Jul 27 17:29:23.366165 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.365877 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9e2569a9-2352-4e32-96b1-fc35089c290e-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-wc86b\" (UID: \"9e2569a9-2352-4e32-96b1-fc35089c290e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wc86b" Jul 27 17:29:23.366165 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.365992 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9e2569a9-2352-4e32-96b1-fc35089c290e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-wc86b\" (UID: \"9e2569a9-2352-4e32-96b1-fc35089c290e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wc86b" Jul 27 17:29:23.366165 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.366017 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e2569a9-2352-4e32-96b1-fc35089c290e-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-wc86b\" (UID: \"9e2569a9-2352-4e32-96b1-fc35089c290e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wc86b" Jul 27 17:29:23.366165 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.366078 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6269d5fd-1774-4aef-9cc7-2296db21f5a3-root\") pod \"node-exporter-h67d4\" (UID: \"6269d5fd-1774-4aef-9cc7-2296db21f5a3\") " pod="openshift-monitoring/node-exporter-h67d4" Jul 27 17:29:23.366165 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:29:23.366137 2570 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 27 17:29:23.366407 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:29:23.366191 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6269d5fd-1774-4aef-9cc7-2296db21f5a3-node-exporter-tls podName:6269d5fd-1774-4aef-9cc7-2296db21f5a3 nodeName:}" failed. No retries permitted until 2026-07-27 17:29:23.866173243 +0000 UTC m=+81.307388198 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/6269d5fd-1774-4aef-9cc7-2296db21f5a3-node-exporter-tls") pod "node-exporter-h67d4" (UID: "6269d5fd-1774-4aef-9cc7-2296db21f5a3") : secret "node-exporter-tls" not found Jul 27 17:29:23.366407 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.366393 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0aee75d-e83d-4d0e-a70e-edea165b58c7-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-cckjt\" (UID: \"a0aee75d-e83d-4d0e-a70e-edea165b58c7\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-cckjt" Jul 27 17:29:23.366581 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.366549 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6269d5fd-1774-4aef-9cc7-2296db21f5a3-metrics-client-ca\") pod \"node-exporter-h67d4\" (UID: \"6269d5fd-1774-4aef-9cc7-2296db21f5a3\") " pod="openshift-monitoring/node-exporter-h67d4" Jul 27 17:29:23.367895 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.367870 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9e2569a9-2352-4e32-96b1-fc35089c290e-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-wc86b\" (UID: \"9e2569a9-2352-4e32-96b1-fc35089c290e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wc86b" Jul 27 17:29:23.368363 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.368340 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a0aee75d-e83d-4d0e-a70e-edea165b58c7-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-cckjt\" (UID: \"a0aee75d-e83d-4d0e-a70e-edea165b58c7\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-cckjt" Jul 27 17:29:23.368952 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.368896 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9e2569a9-2352-4e32-96b1-fc35089c290e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-wc86b\" (UID: \"9e2569a9-2352-4e32-96b1-fc35089c290e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wc86b" Jul 27 17:29:23.369181 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.369163 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6269d5fd-1774-4aef-9cc7-2296db21f5a3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h67d4\" (UID: \"6269d5fd-1774-4aef-9cc7-2296db21f5a3\") " pod="openshift-monitoring/node-exporter-h67d4" Jul 27 17:29:23.369269 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.369257 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a0aee75d-e83d-4d0e-a70e-edea165b58c7-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-cckjt\" (UID: \"a0aee75d-e83d-4d0e-a70e-edea165b58c7\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-cckjt" Jul 27 17:29:23.375482 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.375457 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh6nq\" (UniqueName: \"kubernetes.io/projected/9e2569a9-2352-4e32-96b1-fc35089c290e-kube-api-access-qh6nq\") pod \"kube-state-metrics-56d9f944b4-wc86b\" (UID: \"9e2569a9-2352-4e32-96b1-fc35089c290e\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wc86b" Jul 27 17:29:23.375939 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.375922 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lv7v\" (UniqueName: \"kubernetes.io/projected/a0aee75d-e83d-4d0e-a70e-edea165b58c7-kube-api-access-9lv7v\") pod \"openshift-state-metrics-b8f49f5c6-cckjt\" (UID: \"a0aee75d-e83d-4d0e-a70e-edea165b58c7\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-cckjt" Jul 27 17:29:23.376078 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.376057 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzzsv\" (UniqueName: \"kubernetes.io/projected/6269d5fd-1774-4aef-9cc7-2296db21f5a3-kube-api-access-tzzsv\") pod \"node-exporter-h67d4\" (UID: \"6269d5fd-1774-4aef-9cc7-2296db21f5a3\") " pod="openshift-monitoring/node-exporter-h67d4" Jul 27 17:29:23.484366 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.484277 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-cckjt" Jul 27 17:29:23.508462 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.508429 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wc86b" Jul 27 17:29:23.869039 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.869005 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6269d5fd-1774-4aef-9cc7-2296db21f5a3-node-exporter-tls\") pod \"node-exporter-h67d4\" (UID: \"6269d5fd-1774-4aef-9cc7-2296db21f5a3\") " pod="openshift-monitoring/node-exporter-h67d4" Jul 27 17:29:23.871495 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:23.871466 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6269d5fd-1774-4aef-9cc7-2296db21f5a3-node-exporter-tls\") pod \"node-exporter-h67d4\" (UID: \"6269d5fd-1774-4aef-9cc7-2296db21f5a3\") " pod="openshift-monitoring/node-exporter-h67d4" Jul 27 17:29:24.122905 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:24.122830 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-h67d4" Jul 27 17:29:24.239409 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:24.239372 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 27 17:29:24.262190 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:24.262153 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 27 17:29:24.262359 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:24.262312 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:29:24.264728 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:24.264702 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 27 17:29:24.264870 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:24.264749 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 27 17:29:24.264870 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:24.264707 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 27 17:29:24.264965 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:24.264901 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 27 17:29:24.265110 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:24.265088 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 27 17:29:24.265205 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:24.265184 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 27 17:29:24.265310 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:24.265254 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 27 17:29:24.265534 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:24.265497 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 27 17:29:24.265630 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:24.265559 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 27 17:29:24.265935 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:24.265915 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-fnv5t\"" Jul 27 17:29:24.372796 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:24.372677 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/41e7e826-76ea-4c02-87c8-739b0653b66a-web-config\") pod \"alertmanager-main-0\" (UID: \"41e7e826-76ea-4c02-87c8-739b0653b66a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:29:24.372796 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:24.372735 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/41e7e826-76ea-4c02-87c8-739b0653b66a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"41e7e826-76ea-4c02-87c8-739b0653b66a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:29:24.373027 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:24.372815 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/41e7e826-76ea-4c02-87c8-739b0653b66a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"41e7e826-76ea-4c02-87c8-739b0653b66a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:29:24.373027 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:24.372865 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/41e7e826-76ea-4c02-87c8-739b0653b66a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"41e7e826-76ea-4c02-87c8-739b0653b66a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:29:24.373027 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:24.372898 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/41e7e826-76ea-4c02-87c8-739b0653b66a-config-out\") pod \"alertmanager-main-0\" (UID: \"41e7e826-76ea-4c02-87c8-739b0653b66a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:29:24.373027 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:24.372921 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/41e7e826-76ea-4c02-87c8-739b0653b66a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"41e7e826-76ea-4c02-87c8-739b0653b66a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:29:24.373027 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:24.372984 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41e7e826-76ea-4c02-87c8-739b0653b66a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"41e7e826-76ea-4c02-87c8-739b0653b66a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:29:24.373027 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:24.373020 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qvbj\" (UniqueName: \"kubernetes.io/projected/41e7e826-76ea-4c02-87c8-739b0653b66a-kube-api-access-5qvbj\") pod \"alertmanager-main-0\" (UID: \"41e7e826-76ea-4c02-87c8-739b0653b66a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:29:24.373329 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:24.373047 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/41e7e826-76ea-4c02-87c8-739b0653b66a-config-volume\") pod \"alertmanager-main-0\" (UID: \"41e7e826-76ea-4c02-87c8-739b0653b66a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:29:24.373329 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:24.373088 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/41e7e826-76ea-4c02-87c8-739b0653b66a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"41e7e826-76ea-4c02-87c8-739b0653b66a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:29:24.373329 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:24.373155 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41e7e826-76ea-4c02-87c8-739b0653b66a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"41e7e826-76ea-4c02-87c8-739b0653b66a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:29:24.373329 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:24.373221 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/41e7e826-76ea-4c02-87c8-739b0653b66a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"41e7e826-76ea-4c02-87c8-739b0653b66a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:29:24.373329 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:24.373251 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/41e7e826-76ea-4c02-87c8-739b0653b66a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"41e7e826-76ea-4c02-87c8-739b0653b66a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:29:24.473881 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:24.473838 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41e7e826-76ea-4c02-87c8-739b0653b66a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"41e7e826-76ea-4c02-87c8-739b0653b66a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:29:24.473881 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:24.473886 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qvbj\" (UniqueName: \"kubernetes.io/projected/41e7e826-76ea-4c02-87c8-739b0653b66a-kube-api-access-5qvbj\") pod \"alertmanager-main-0\" (UID: \"41e7e826-76ea-4c02-87c8-739b0653b66a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:29:24.474153 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:24.473909 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/41e7e826-76ea-4c02-87c8-739b0653b66a-config-volume\") pod \"alertmanager-main-0\" (UID: \"41e7e826-76ea-4c02-87c8-739b0653b66a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:29:24.474153 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:24.473951 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/41e7e826-76ea-4c02-87c8-739b0653b66a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"41e7e826-76ea-4c02-87c8-739b0653b66a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:29:24.474257 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:24.474165 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41e7e826-76ea-4c02-87c8-739b0653b66a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"41e7e826-76ea-4c02-87c8-739b0653b66a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:29:24.474257 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:24.474208 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/41e7e826-76ea-4c02-87c8-739b0653b66a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"41e7e826-76ea-4c02-87c8-739b0653b66a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:29:24.474257 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:24.474242 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/41e7e826-76ea-4c02-87c8-739b0653b66a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"41e7e826-76ea-4c02-87c8-739b0653b66a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:29:24.474423 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:24.474392 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/41e7e826-76ea-4c02-87c8-739b0653b66a-web-config\") pod \"alertmanager-main-0\" (UID: \"41e7e826-76ea-4c02-87c8-739b0653b66a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:29:24.474501 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:24.474481 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/41e7e826-76ea-4c02-87c8-739b0653b66a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"41e7e826-76ea-4c02-87c8-739b0653b66a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:29:24.474638 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:29:24.474608 2570 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 27 17:29:24.474704 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:24.474636 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/41e7e826-76ea-4c02-87c8-739b0653b66a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"41e7e826-76ea-4c02-87c8-739b0653b66a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:29:24.474704 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:29:24.474672 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41e7e826-76ea-4c02-87c8-739b0653b66a-secret-alertmanager-main-tls podName:41e7e826-76ea-4c02-87c8-739b0653b66a nodeName:}" failed. No retries permitted until 2026-07-27 17:29:24.974651525 +0000 UTC m=+82.415866481 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/41e7e826-76ea-4c02-87c8-739b0653b66a-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "41e7e826-76ea-4c02-87c8-739b0653b66a") : secret "alertmanager-main-tls" not found Jul 27 17:29:24.474816 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:24.474724 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/41e7e826-76ea-4c02-87c8-739b0653b66a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"41e7e826-76ea-4c02-87c8-739b0653b66a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:29:24.474816 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:24.474758 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/41e7e826-76ea-4c02-87c8-739b0653b66a-config-out\") pod \"alertmanager-main-0\" (UID: \"41e7e826-76ea-4c02-87c8-739b0653b66a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:29:24.474816 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:24.474782 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/41e7e826-76ea-4c02-87c8-739b0653b66a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"41e7e826-76ea-4c02-87c8-739b0653b66a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:29:24.475250 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:24.475000 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41e7e826-76ea-4c02-87c8-739b0653b66a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"41e7e826-76ea-4c02-87c8-739b0653b66a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:29:24.475250 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:24.475152 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41e7e826-76ea-4c02-87c8-739b0653b66a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"41e7e826-76ea-4c02-87c8-739b0653b66a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:29:24.475250 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:24.475152 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/41e7e826-76ea-4c02-87c8-739b0653b66a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"41e7e826-76ea-4c02-87c8-739b0653b66a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:29:24.477648 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:24.477591 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/41e7e826-76ea-4c02-87c8-739b0653b66a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"41e7e826-76ea-4c02-87c8-739b0653b66a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:29:24.477760 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:24.477679 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/41e7e826-76ea-4c02-87c8-739b0653b66a-config-out\") pod \"alertmanager-main-0\" (UID: \"41e7e826-76ea-4c02-87c8-739b0653b66a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:29:24.477949 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:24.477929 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/41e7e826-76ea-4c02-87c8-739b0653b66a-web-config\") pod \"alertmanager-main-0\" (UID: \"41e7e826-76ea-4c02-87c8-739b0653b66a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:29:24.478600 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:24.478577 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/41e7e826-76ea-4c02-87c8-739b0653b66a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"41e7e826-76ea-4c02-87c8-739b0653b66a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:29:24.478702 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:24.478683 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/41e7e826-76ea-4c02-87c8-739b0653b66a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"41e7e826-76ea-4c02-87c8-739b0653b66a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:29:24.479326 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:24.479289 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/41e7e826-76ea-4c02-87c8-739b0653b66a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"41e7e826-76ea-4c02-87c8-739b0653b66a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:29:24.479422 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:24.479356 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/41e7e826-76ea-4c02-87c8-739b0653b66a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"41e7e826-76ea-4c02-87c8-739b0653b66a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:29:24.480266 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:24.480244 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/41e7e826-76ea-4c02-87c8-739b0653b66a-config-volume\") pod \"alertmanager-main-0\" (UID: \"41e7e826-76ea-4c02-87c8-739b0653b66a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:29:24.482170 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:24.482150 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qvbj\" (UniqueName: \"kubernetes.io/projected/41e7e826-76ea-4c02-87c8-739b0653b66a-kube-api-access-5qvbj\") pod \"alertmanager-main-0\" (UID: \"41e7e826-76ea-4c02-87c8-739b0653b66a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:29:24.978609 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:24.978571 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/41e7e826-76ea-4c02-87c8-739b0653b66a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"41e7e826-76ea-4c02-87c8-739b0653b66a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:29:24.981330 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:24.981307 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/41e7e826-76ea-4c02-87c8-739b0653b66a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"41e7e826-76ea-4c02-87c8-739b0653b66a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:29:25.174406 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:25.174364 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:29:25.940555 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:25.940505 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-675f4b885b-bggck"] Jul 27 17:29:27.080986 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:29:27.080941 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6269d5fd_1774_4aef_9cc7_2296db21f5a3.slice/crio-b85210ee762fb4039bbde0bb7da7f4e1518657144f9d9cdc6f97aa0e8bae0bd3 WatchSource:0}: Error finding container b85210ee762fb4039bbde0bb7da7f4e1518657144f9d9cdc6f97aa0e8bae0bd3: Status 404 returned error can't find the container with id b85210ee762fb4039bbde0bb7da7f4e1518657144f9d9cdc6f97aa0e8bae0bd3 Jul 27 17:29:27.249840 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:27.249742 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-wc86b"] Jul 27 17:29:27.251870 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:29:27.251839 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e2569a9_2352_4e32_96b1_fc35089c290e.slice/crio-2c10cdb09ae92774eeb7134d1a3c7bba3b6f24b4ab397adf1f1c0d1a4102c9a8 WatchSource:0}: Error finding container 2c10cdb09ae92774eeb7134d1a3c7bba3b6f24b4ab397adf1f1c0d1a4102c9a8: Status 404 returned error can't find the container with id 2c10cdb09ae92774eeb7134d1a3c7bba3b6f24b4ab397adf1f1c0d1a4102c9a8 Jul 27 17:29:27.263471 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:27.263383 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-cckjt"] Jul 27 17:29:27.268443 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:29:27.268412 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0aee75d_e83d_4d0e_a70e_edea165b58c7.slice/crio-c2923e0f7a3d80868532d8c7962200707a745212db529d7e0c80e264cdb0bc78 WatchSource:0}: Error finding container c2923e0f7a3d80868532d8c7962200707a745212db529d7e0c80e264cdb0bc78: Status 404 returned error can't find the container with id c2923e0f7a3d80868532d8c7962200707a745212db529d7e0c80e264cdb0bc78 Jul 27 17:29:27.289979 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:27.289941 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 27 17:29:27.294454 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:29:27.294421 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41e7e826_76ea_4c02_87c8_739b0653b66a.slice/crio-840eb68207b058db9ef29ff451726d5c11e295b4d0b846e39da35cc53482dc77 WatchSource:0}: Error finding container 840eb68207b058db9ef29ff451726d5c11e295b4d0b846e39da35cc53482dc77: Status 404 returned error can't find the container with id 840eb68207b058db9ef29ff451726d5c11e295b4d0b846e39da35cc53482dc77 Jul 27 17:29:27.416381 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:27.416361 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6bccbd7d69-hmc99" Jul 27 17:29:27.529061 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:27.529021 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h67d4" event={"ID":"6269d5fd-1774-4aef-9cc7-2296db21f5a3","Type":"ContainerStarted","Data":"b85210ee762fb4039bbde0bb7da7f4e1518657144f9d9cdc6f97aa0e8bae0bd3"} Jul 27 17:29:27.530641 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:27.530617 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-675f4b885b-bggck" event={"ID":"f1389cc5-47f4-4804-8c4e-836a5181bd76","Type":"ContainerStarted","Data":"023762b8b11e1512ee5d600c7ac6da17813cb9efc5b5e6c8990ae589913788ba"} Jul 27 17:29:27.531936 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:27.531910 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"41e7e826-76ea-4c02-87c8-739b0653b66a","Type":"ContainerStarted","Data":"840eb68207b058db9ef29ff451726d5c11e295b4d0b846e39da35cc53482dc77"} Jul 27 17:29:27.533673 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:27.533649 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-cckjt" event={"ID":"a0aee75d-e83d-4d0e-a70e-edea165b58c7","Type":"ContainerStarted","Data":"06bf8e217badc4e5a372306a166f57985553c6f569cc26f8c0efbe136c401e0b"} Jul 27 17:29:27.533761 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:27.533694 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-cckjt" event={"ID":"a0aee75d-e83d-4d0e-a70e-edea165b58c7","Type":"ContainerStarted","Data":"f6241a811e64a15df8e1bb8bdf2d0ee000eca51617c1172a11a72f9603b71d41"} Jul 27 17:29:27.533761 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:27.533713 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-cckjt" event={"ID":"a0aee75d-e83d-4d0e-a70e-edea165b58c7","Type":"ContainerStarted","Data":"c2923e0f7a3d80868532d8c7962200707a745212db529d7e0c80e264cdb0bc78"} Jul 27 17:29:27.534831 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:27.534810 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wc86b" event={"ID":"9e2569a9-2352-4e32-96b1-fc35089c290e","Type":"ContainerStarted","Data":"2c10cdb09ae92774eeb7134d1a3c7bba3b6f24b4ab397adf1f1c0d1a4102c9a8"} Jul 27 17:29:27.536185 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:27.536164 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-bgt4v" event={"ID":"ad4401c9-a50e-46d6-a1f6-e4d8bf591064","Type":"ContainerStarted","Data":"22cb2fead841e5be8b026e5cc02a3d2751ab6079afcc4441fdd05d8085929fe0"} Jul 27 17:29:27.536379 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:27.536362 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-bgt4v" Jul 27 17:29:27.537672 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:27.537650 2570 patch_prober.go:28] interesting pod/downloads-7c97dc4555-bgt4v container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.133.0.10:8080/\": dial tcp 10.133.0.10:8080: connect: connection refused" start-of-body= Jul 27 17:29:27.537765 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:27.537699 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-7c97dc4555-bgt4v" podUID="ad4401c9-a50e-46d6-a1f6-e4d8bf591064" containerName="download-server" probeResult="failure" output="Get \"http://10.133.0.10:8080/\": dial tcp 10.133.0.10:8080: connect: connection refused" Jul 27 17:29:27.590858 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:27.590771 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-675f4b885b-bggck" podStartSLOduration=4.705869762 podStartE2EDuration="14.590751165s" totalImagesPullingTime="9.884881403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:29:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:29:27.590317021 +0000 UTC m=+85.031531995" watchObservedRunningTime="2026-07-27 17:29:27.590751165 +0000 UTC m=+85.031966139" Jul 27 17:29:27.629497 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:27.626005 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-bgt4v" podStartSLOduration=1.659251056 podStartE2EDuration="23.625983892s" totalImagesPullingTime="21.966732836s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:29:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:29:27.624007722 +0000 UTC m=+85.065222697" watchObservedRunningTime="2026-07-27 17:29:27.625983892 +0000 UTC m=+85.067198869" Jul 27 17:29:27.942726 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:27.942596 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-gbk2t"] Jul 27 17:29:27.964285 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:27.963206 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-gbk2t"] Jul 27 17:29:27.964285 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:27.963341 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-gbk2t" Jul 27 17:29:27.965874 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:27.965838 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-7qlp7\"" Jul 27 17:29:27.966138 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:27.966084 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 27 17:29:28.007331 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:28.007175 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqklc\" (UniqueName: \"kubernetes.io/projected/d1faee83-1b22-4bb7-b51d-06f34785dc23-kube-api-access-lqklc\") pod \"monitoring-plugin-68b7fc9574-gbk2t\" (UID: \"d1faee83-1b22-4bb7-b51d-06f34785dc23\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-gbk2t" Jul 27 17:29:28.007331 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:28.007236 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d1faee83-1b22-4bb7-b51d-06f34785dc23-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-gbk2t\" (UID: \"d1faee83-1b22-4bb7-b51d-06f34785dc23\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-gbk2t" Jul 27 17:29:28.108223 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:28.108131 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d1faee83-1b22-4bb7-b51d-06f34785dc23-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-gbk2t\" (UID: \"d1faee83-1b22-4bb7-b51d-06f34785dc23\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-gbk2t" Jul 27 17:29:28.109178 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:28.108266 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqklc\" (UniqueName: \"kubernetes.io/projected/d1faee83-1b22-4bb7-b51d-06f34785dc23-kube-api-access-lqklc\") pod \"monitoring-plugin-68b7fc9574-gbk2t\" (UID: \"d1faee83-1b22-4bb7-b51d-06f34785dc23\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-gbk2t" Jul 27 17:29:28.109178 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:29:28.108654 2570 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 27 17:29:28.109178 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:29:28.108721 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1faee83-1b22-4bb7-b51d-06f34785dc23-monitoring-plugin-cert podName:d1faee83-1b22-4bb7-b51d-06f34785dc23 nodeName:}" failed. No retries permitted until 2026-07-27 17:29:28.608702379 +0000 UTC m=+86.049917354 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/d1faee83-1b22-4bb7-b51d-06f34785dc23-monitoring-plugin-cert") pod "monitoring-plugin-68b7fc9574-gbk2t" (UID: "d1faee83-1b22-4bb7-b51d-06f34785dc23") : secret "monitoring-plugin-cert" not found Jul 27 17:29:28.120933 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:28.120879 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqklc\" (UniqueName: \"kubernetes.io/projected/d1faee83-1b22-4bb7-b51d-06f34785dc23-kube-api-access-lqklc\") pod \"monitoring-plugin-68b7fc9574-gbk2t\" (UID: \"d1faee83-1b22-4bb7-b51d-06f34785dc23\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-gbk2t" Jul 27 17:29:28.560063 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:28.559945 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-bgt4v" Jul 27 17:29:28.612450 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:28.612418 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d1faee83-1b22-4bb7-b51d-06f34785dc23-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-gbk2t\" (UID: \"d1faee83-1b22-4bb7-b51d-06f34785dc23\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-gbk2t" Jul 27 17:29:28.615218 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:28.615191 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d1faee83-1b22-4bb7-b51d-06f34785dc23-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-gbk2t\" (UID: \"d1faee83-1b22-4bb7-b51d-06f34785dc23\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-gbk2t" Jul 27 17:29:28.877739 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:28.877588 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-gbk2t" Jul 27 17:29:29.593896 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:29.592485 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7848d54945-sg2cv"] Jul 27 17:29:29.614430 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:29.614231 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7848d54945-sg2cv"] Jul 27 17:29:29.614430 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:29.614357 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7848d54945-sg2cv" Jul 27 17:29:29.624456 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:29.624433 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 27 17:29:29.728258 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:29.726070 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c3629ded-e6b6-4068-9490-484e343b7a38-oauth-serving-cert\") pod \"console-7848d54945-sg2cv\" (UID: \"c3629ded-e6b6-4068-9490-484e343b7a38\") " pod="openshift-console/console-7848d54945-sg2cv" Jul 27 17:29:29.728258 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:29.726157 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd9wl\" (UniqueName: \"kubernetes.io/projected/c3629ded-e6b6-4068-9490-484e343b7a38-kube-api-access-dd9wl\") pod \"console-7848d54945-sg2cv\" (UID: \"c3629ded-e6b6-4068-9490-484e343b7a38\") " pod="openshift-console/console-7848d54945-sg2cv" Jul 27 17:29:29.728258 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:29.726207 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c3629ded-e6b6-4068-9490-484e343b7a38-console-oauth-config\") pod \"console-7848d54945-sg2cv\" (UID: \"c3629ded-e6b6-4068-9490-484e343b7a38\") " pod="openshift-console/console-7848d54945-sg2cv" Jul 27 17:29:29.728258 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:29.726237 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c3629ded-e6b6-4068-9490-484e343b7a38-service-ca\") pod \"console-7848d54945-sg2cv\" (UID: \"c3629ded-e6b6-4068-9490-484e343b7a38\") " pod="openshift-console/console-7848d54945-sg2cv" Jul 27 17:29:29.728258 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:29.726278 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3629ded-e6b6-4068-9490-484e343b7a38-trusted-ca-bundle\") pod \"console-7848d54945-sg2cv\" (UID: \"c3629ded-e6b6-4068-9490-484e343b7a38\") " pod="openshift-console/console-7848d54945-sg2cv" Jul 27 17:29:29.728258 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:29.726312 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c3629ded-e6b6-4068-9490-484e343b7a38-console-serving-cert\") pod \"console-7848d54945-sg2cv\" (UID: \"c3629ded-e6b6-4068-9490-484e343b7a38\") " pod="openshift-console/console-7848d54945-sg2cv" Jul 27 17:29:29.728258 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:29.726339 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c3629ded-e6b6-4068-9490-484e343b7a38-console-config\") pod \"console-7848d54945-sg2cv\" (UID: \"c3629ded-e6b6-4068-9490-484e343b7a38\") " pod="openshift-console/console-7848d54945-sg2cv" Jul 27 17:29:29.793309 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:29.793220 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-gbk2t"] Jul 27 17:29:29.830014 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:29.827849 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c3629ded-e6b6-4068-9490-484e343b7a38-oauth-serving-cert\") pod \"console-7848d54945-sg2cv\" (UID: \"c3629ded-e6b6-4068-9490-484e343b7a38\") " pod="openshift-console/console-7848d54945-sg2cv" Jul 27 17:29:29.830014 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:29.827939 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd9wl\" (UniqueName: \"kubernetes.io/projected/c3629ded-e6b6-4068-9490-484e343b7a38-kube-api-access-dd9wl\") pod \"console-7848d54945-sg2cv\" (UID: \"c3629ded-e6b6-4068-9490-484e343b7a38\") " pod="openshift-console/console-7848d54945-sg2cv" Jul 27 17:29:29.830014 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:29.827997 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c3629ded-e6b6-4068-9490-484e343b7a38-console-oauth-config\") pod \"console-7848d54945-sg2cv\" (UID: \"c3629ded-e6b6-4068-9490-484e343b7a38\") " pod="openshift-console/console-7848d54945-sg2cv" Jul 27 17:29:29.830014 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:29.828032 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c3629ded-e6b6-4068-9490-484e343b7a38-service-ca\") pod \"console-7848d54945-sg2cv\" (UID: \"c3629ded-e6b6-4068-9490-484e343b7a38\") " pod="openshift-console/console-7848d54945-sg2cv" Jul 27 17:29:29.830014 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:29.828072 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3629ded-e6b6-4068-9490-484e343b7a38-trusted-ca-bundle\") pod \"console-7848d54945-sg2cv\" (UID: \"c3629ded-e6b6-4068-9490-484e343b7a38\") " pod="openshift-console/console-7848d54945-sg2cv" Jul 27 17:29:29.830014 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:29.828103 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c3629ded-e6b6-4068-9490-484e343b7a38-console-serving-cert\") pod \"console-7848d54945-sg2cv\" (UID: \"c3629ded-e6b6-4068-9490-484e343b7a38\") " pod="openshift-console/console-7848d54945-sg2cv" Jul 27 17:29:29.830014 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:29.828127 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c3629ded-e6b6-4068-9490-484e343b7a38-console-config\") pod \"console-7848d54945-sg2cv\" (UID: \"c3629ded-e6b6-4068-9490-484e343b7a38\") " pod="openshift-console/console-7848d54945-sg2cv" Jul 27 17:29:29.830014 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:29.828949 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c3629ded-e6b6-4068-9490-484e343b7a38-console-config\") pod \"console-7848d54945-sg2cv\" (UID: \"c3629ded-e6b6-4068-9490-484e343b7a38\") " pod="openshift-console/console-7848d54945-sg2cv" Jul 27 17:29:29.830014 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:29.829557 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c3629ded-e6b6-4068-9490-484e343b7a38-oauth-serving-cert\") pod \"console-7848d54945-sg2cv\" (UID: \"c3629ded-e6b6-4068-9490-484e343b7a38\") " pod="openshift-console/console-7848d54945-sg2cv" Jul 27 17:29:29.832069 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:29.831117 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c3629ded-e6b6-4068-9490-484e343b7a38-service-ca\") pod \"console-7848d54945-sg2cv\" (UID: \"c3629ded-e6b6-4068-9490-484e343b7a38\") " pod="openshift-console/console-7848d54945-sg2cv" Jul 27 17:29:29.832069 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:29.831925 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3629ded-e6b6-4068-9490-484e343b7a38-trusted-ca-bundle\") pod \"console-7848d54945-sg2cv\" (UID: \"c3629ded-e6b6-4068-9490-484e343b7a38\") " pod="openshift-console/console-7848d54945-sg2cv" Jul 27 17:29:29.835068 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:29.835042 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c3629ded-e6b6-4068-9490-484e343b7a38-console-oauth-config\") pod \"console-7848d54945-sg2cv\" (UID: \"c3629ded-e6b6-4068-9490-484e343b7a38\") " pod="openshift-console/console-7848d54945-sg2cv" Jul 27 17:29:29.835755 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:29.835712 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c3629ded-e6b6-4068-9490-484e343b7a38-console-serving-cert\") pod \"console-7848d54945-sg2cv\" (UID: \"c3629ded-e6b6-4068-9490-484e343b7a38\") " pod="openshift-console/console-7848d54945-sg2cv" Jul 27 17:29:29.839044 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:29.838996 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd9wl\" (UniqueName: \"kubernetes.io/projected/c3629ded-e6b6-4068-9490-484e343b7a38-kube-api-access-dd9wl\") pod \"console-7848d54945-sg2cv\" (UID: \"c3629ded-e6b6-4068-9490-484e343b7a38\") " pod="openshift-console/console-7848d54945-sg2cv" Jul 27 17:29:29.946047 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:29.945812 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7848d54945-sg2cv" Jul 27 17:29:30.188661 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:30.188593 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7848d54945-sg2cv"] Jul 27 17:29:30.459924 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:29:30.459822 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3629ded_e6b6_4068_9490_484e343b7a38.slice/crio-cef1cdb150b7d3d6944d664ca4668e6448406270d88f6f522fe77506ce7d6059 WatchSource:0}: Error finding container cef1cdb150b7d3d6944d664ca4668e6448406270d88f6f522fe77506ce7d6059: Status 404 returned error can't find the container with id cef1cdb150b7d3d6944d664ca4668e6448406270d88f6f522fe77506ce7d6059 Jul 27 17:29:30.559594 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:30.559557 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-gbk2t" event={"ID":"d1faee83-1b22-4bb7-b51d-06f34785dc23","Type":"ContainerStarted","Data":"cf80fa111ef026da4fffed38adecb1dfbec03f09b53da737f3a46f0d47498753"} Jul 27 17:29:30.563944 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:30.563673 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-cckjt" event={"ID":"a0aee75d-e83d-4d0e-a70e-edea165b58c7","Type":"ContainerStarted","Data":"721f71ac795a5bdcdbb1c3a3fea025decb336e4f3a5dcb953bbc8681c5744cbf"} Jul 27 17:29:30.567291 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:30.567247 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wc86b" event={"ID":"9e2569a9-2352-4e32-96b1-fc35089c290e","Type":"ContainerStarted","Data":"43560e58ad113a0581621da9d5f380ca01ae6562857495537cac7e0f8c64b33f"} Jul 27 17:29:30.567426 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:30.567316 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wc86b" event={"ID":"9e2569a9-2352-4e32-96b1-fc35089c290e","Type":"ContainerStarted","Data":"ecffbe6a77766d9ef68412a63a2e7459c9f3d34bde327b45c911d9a8412504fd"} Jul 27 17:29:30.567426 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:30.567345 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wc86b" event={"ID":"9e2569a9-2352-4e32-96b1-fc35089c290e","Type":"ContainerStarted","Data":"d655a800085ee4e63534e63ab3ddde122da105cd32f505ec2674f4e87cee8e73"} Jul 27 17:29:30.570159 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:30.570131 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7848d54945-sg2cv" event={"ID":"c3629ded-e6b6-4068-9490-484e343b7a38","Type":"ContainerStarted","Data":"cef1cdb150b7d3d6944d664ca4668e6448406270d88f6f522fe77506ce7d6059"} Jul 27 17:29:30.586988 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:30.586933 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-cckjt" podStartSLOduration=5.460252532 podStartE2EDuration="7.586914996s" totalImagesPullingTime="2.126662464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:29:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:29:30.585707258 +0000 UTC m=+88.026922233" watchObservedRunningTime="2026-07-27 17:29:30.586914996 +0000 UTC m=+88.028129971" Jul 27 17:29:30.605433 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:30.605379 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wc86b" podStartSLOduration=5.247028566 podStartE2EDuration="7.605361815s" totalImagesPullingTime="2.358333249s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:29:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:29:30.605019556 +0000 UTC m=+88.046234530" watchObservedRunningTime="2026-07-27 17:29:30.605361815 +0000 UTC m=+88.046576788" Jul 27 17:29:32.581222 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:32.581188 2570 generic.go:356] "Generic (PLEG): container finished" podID="6269d5fd-1774-4aef-9cc7-2296db21f5a3" containerID="d3a8e5249c0c54c69d7b7dfd7e76fdae013b58862ee09b4dc31f2b19970e60e5" exitCode=0 Jul 27 17:29:32.581751 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:32.581288 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h67d4" event={"ID":"6269d5fd-1774-4aef-9cc7-2296db21f5a3","Type":"ContainerDied","Data":"d3a8e5249c0c54c69d7b7dfd7e76fdae013b58862ee09b4dc31f2b19970e60e5"} Jul 27 17:29:32.583168 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:32.583140 2570 generic.go:356] "Generic (PLEG): container finished" podID="41e7e826-76ea-4c02-87c8-739b0653b66a" containerID="34daf9820eb485de580b293272dfc9ba260df4f93634def819a070074830afe9" exitCode=0 Jul 27 17:29:32.583290 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:32.583234 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"41e7e826-76ea-4c02-87c8-739b0653b66a","Type":"ContainerDied","Data":"34daf9820eb485de580b293272dfc9ba260df4f93634def819a070074830afe9"} Jul 27 17:29:32.585189 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:32.585162 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7848d54945-sg2cv" event={"ID":"c3629ded-e6b6-4068-9490-484e343b7a38","Type":"ContainerStarted","Data":"d5115a45e56a7747a324feb4d933f0b7e8b6e6f36af7b6f1d9e7b08b974ae761"} Jul 27 17:29:32.586974 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:32.586948 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-gbk2t" event={"ID":"d1faee83-1b22-4bb7-b51d-06f34785dc23","Type":"ContainerStarted","Data":"8c29615877456156b3f9f3448502dc231146d3459f09a9e59c1ab6f385c62409"} Jul 27 17:29:32.587214 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:32.587170 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-gbk2t" Jul 27 17:29:32.592849 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:32.592826 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-gbk2t" Jul 27 17:29:32.648393 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:32.648242 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-gbk2t" podStartSLOduration=3.279565683 podStartE2EDuration="5.648219616s" totalImagesPullingTime="2.368653933s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:29:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:29:32.646405589 +0000 UTC m=+90.087620563" watchObservedRunningTime="2026-07-27 17:29:32.648219616 +0000 UTC m=+90.089434590" Jul 27 17:29:32.673338 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:32.673279 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7848d54945-sg2cv" podStartSLOduration=3.673259447 podStartE2EDuration="3.673259447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:29:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:29:32.671416052 +0000 UTC m=+90.112631027" watchObservedRunningTime="2026-07-27 17:29:32.673259447 +0000 UTC m=+90.114474423" Jul 27 17:29:33.597600 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:33.597472 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h67d4" event={"ID":"6269d5fd-1774-4aef-9cc7-2296db21f5a3","Type":"ContainerStarted","Data":"1d8ba0a2d02ff9dd0e8d4d101fa8d11a3461d0e2653ddc7a3b484a32fbfdcf87"} Jul 27 17:29:33.598079 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:33.597620 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h67d4" event={"ID":"6269d5fd-1774-4aef-9cc7-2296db21f5a3","Type":"ContainerStarted","Data":"8591522090f4696dedfb289544a08e20ff65f44c002283840476155f86bf7afa"} Jul 27 17:29:33.631126 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:33.631043 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-h67d4" podStartSLOduration=6.412460562 podStartE2EDuration="10.631021656s" totalImagesPullingTime="4.110771611s" totalInitContainerRuntime="107.789483ms" isStatefulPod=true podCreationTimestamp="2026-07-27 17:29:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:29:33.630880234 +0000 UTC m=+91.072095210" watchObservedRunningTime="2026-07-27 17:29:33.631021656 +0000 UTC m=+91.072236630" Jul 27 17:29:34.289001 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:34.288966 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-675f4b885b-bggck" Jul 27 17:29:34.605882 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:34.605844 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"41e7e826-76ea-4c02-87c8-739b0653b66a","Type":"ContainerStarted","Data":"c3e0f580e654ecba2ebb2afb155e162612da3b7897a5f19620316e86819c0148"} Jul 27 17:29:35.613446 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:35.613409 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"41e7e826-76ea-4c02-87c8-739b0653b66a","Type":"ContainerStarted","Data":"d4a3b4130cbd345d25d484bb2b561d67c83a40fa95d77948cf22d9f401c4c855"} Jul 27 17:29:35.613860 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:35.613487 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"41e7e826-76ea-4c02-87c8-739b0653b66a","Type":"ContainerStarted","Data":"b2f4898e0a1243e52057a1a66f0cd0f5cf35858fdb46c56201ffa0323eecb983"} Jul 27 17:29:35.613860 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:35.613554 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"41e7e826-76ea-4c02-87c8-739b0653b66a","Type":"ContainerStarted","Data":"5af24472f29983b8490f501232ef466f348cc8c6103af90d4c1c5133bcdeb8df"} Jul 27 17:29:35.613860 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:35.613595 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"41e7e826-76ea-4c02-87c8-739b0653b66a","Type":"ContainerStarted","Data":"322d221f68a0492ab95876071778dd8cecb0ab5bde0cdf9f8ce606dfaa2cfc18"} Jul 27 17:29:37.624647 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:37.624611 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"41e7e826-76ea-4c02-87c8-739b0653b66a","Type":"ContainerStarted","Data":"7349ab94536e9a73fc37447f4efa62b7c843f3b0dca3c56b83351744cf69888a"} Jul 27 17:29:37.651105 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:37.651055 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=5.50279526 podStartE2EDuration="13.651040029s" totalImagesPullingTime="8.120988635s" totalInitContainerRuntime="27.256134ms" isStatefulPod=false podCreationTimestamp="2026-07-27 17:29:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:29:37.650257604 +0000 UTC m=+95.091472611" watchObservedRunningTime="2026-07-27 17:29:37.651040029 +0000 UTC m=+95.092255002" Jul 27 17:29:39.946756 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:39.946718 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7848d54945-sg2cv" Jul 27 17:29:39.947321 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:39.946783 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7848d54945-sg2cv" Jul 27 17:29:39.951553 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:39.951508 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7848d54945-sg2cv" Jul 27 17:29:40.639604 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:40.639562 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7848d54945-sg2cv" Jul 27 17:29:48.497660 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:48.497618 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-jhzgt" Jul 27 17:29:52.550423 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:52.550365 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-675f4b885b-bggck" podUID="f1389cc5-47f4-4804-8c4e-836a5181bd76" containerName="console" containerID="cri-o://023762b8b11e1512ee5d600c7ac6da17813cb9efc5b5e6c8990ae589913788ba" gracePeriod=15 Jul 27 17:29:52.680823 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:52.680785 2570 generic.go:356] "Generic (PLEG): container finished" podID="f1389cc5-47f4-4804-8c4e-836a5181bd76" containerID="023762b8b11e1512ee5d600c7ac6da17813cb9efc5b5e6c8990ae589913788ba" exitCode=0 Jul 27 17:29:52.680980 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:52.680855 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-675f4b885b-bggck" event={"ID":"f1389cc5-47f4-4804-8c4e-836a5181bd76","Type":"ContainerDied","Data":"023762b8b11e1512ee5d600c7ac6da17813cb9efc5b5e6c8990ae589913788ba"} Jul 27 17:29:52.817043 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:52.816971 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-675f4b885b-bggck" Jul 27 17:29:52.834254 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:52.834230 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f1389cc5-47f4-4804-8c4e-836a5181bd76-service-ca\") pod \"f1389cc5-47f4-4804-8c4e-836a5181bd76\" (UID: \"f1389cc5-47f4-4804-8c4e-836a5181bd76\") " Jul 27 17:29:52.834359 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:52.834261 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f1389cc5-47f4-4804-8c4e-836a5181bd76-oauth-serving-cert\") pod \"f1389cc5-47f4-4804-8c4e-836a5181bd76\" (UID: \"f1389cc5-47f4-4804-8c4e-836a5181bd76\") " Jul 27 17:29:52.834359 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:52.834285 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-42wcp\" (UniqueName: \"kubernetes.io/projected/f1389cc5-47f4-4804-8c4e-836a5181bd76-kube-api-access-42wcp\") pod \"f1389cc5-47f4-4804-8c4e-836a5181bd76\" (UID: \"f1389cc5-47f4-4804-8c4e-836a5181bd76\") " Jul 27 17:29:52.834359 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:52.834315 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f1389cc5-47f4-4804-8c4e-836a5181bd76-console-oauth-config\") pod \"f1389cc5-47f4-4804-8c4e-836a5181bd76\" (UID: \"f1389cc5-47f4-4804-8c4e-836a5181bd76\") " Jul 27 17:29:52.834359 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:52.834355 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f1389cc5-47f4-4804-8c4e-836a5181bd76-console-serving-cert\") pod \"f1389cc5-47f4-4804-8c4e-836a5181bd76\" (UID: \"f1389cc5-47f4-4804-8c4e-836a5181bd76\") " Jul 27 17:29:52.834543 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:52.834395 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f1389cc5-47f4-4804-8c4e-836a5181bd76-console-config\") pod \"f1389cc5-47f4-4804-8c4e-836a5181bd76\" (UID: \"f1389cc5-47f4-4804-8c4e-836a5181bd76\") " Jul 27 17:29:52.834805 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:52.834779 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1389cc5-47f4-4804-8c4e-836a5181bd76-service-ca" (OuterVolumeSpecName: "service-ca") pod "f1389cc5-47f4-4804-8c4e-836a5181bd76" (UID: "f1389cc5-47f4-4804-8c4e-836a5181bd76"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:29:52.834903 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:52.834872 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1389cc5-47f4-4804-8c4e-836a5181bd76-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f1389cc5-47f4-4804-8c4e-836a5181bd76" (UID: "f1389cc5-47f4-4804-8c4e-836a5181bd76"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:29:52.834903 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:52.834877 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1389cc5-47f4-4804-8c4e-836a5181bd76-console-config" (OuterVolumeSpecName: "console-config") pod "f1389cc5-47f4-4804-8c4e-836a5181bd76" (UID: "f1389cc5-47f4-4804-8c4e-836a5181bd76"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:29:52.836604 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:52.836583 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1389cc5-47f4-4804-8c4e-836a5181bd76-kube-api-access-42wcp" (OuterVolumeSpecName: "kube-api-access-42wcp") pod "f1389cc5-47f4-4804-8c4e-836a5181bd76" (UID: "f1389cc5-47f4-4804-8c4e-836a5181bd76"). InnerVolumeSpecName "kube-api-access-42wcp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:29:52.836946 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:52.836930 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1389cc5-47f4-4804-8c4e-836a5181bd76-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f1389cc5-47f4-4804-8c4e-836a5181bd76" (UID: "f1389cc5-47f4-4804-8c4e-836a5181bd76"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 17:29:52.837041 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:52.837018 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1389cc5-47f4-4804-8c4e-836a5181bd76-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f1389cc5-47f4-4804-8c4e-836a5181bd76" (UID: "f1389cc5-47f4-4804-8c4e-836a5181bd76"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 17:29:52.935643 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:52.935608 2570 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f1389cc5-47f4-4804-8c4e-836a5181bd76-console-config\") on node \"ip-10-0-140-185.ec2.internal\" DevicePath \"\"" Jul 27 17:29:52.935643 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:52.935639 2570 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f1389cc5-47f4-4804-8c4e-836a5181bd76-service-ca\") on node \"ip-10-0-140-185.ec2.internal\" DevicePath \"\"" Jul 27 17:29:52.935643 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:52.935650 2570 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f1389cc5-47f4-4804-8c4e-836a5181bd76-oauth-serving-cert\") on node \"ip-10-0-140-185.ec2.internal\" DevicePath \"\"" Jul 27 17:29:52.935895 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:52.935659 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42wcp\" (UniqueName: \"kubernetes.io/projected/f1389cc5-47f4-4804-8c4e-836a5181bd76-kube-api-access-42wcp\") on node \"ip-10-0-140-185.ec2.internal\" DevicePath \"\"" Jul 27 17:29:52.935895 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:52.935668 2570 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f1389cc5-47f4-4804-8c4e-836a5181bd76-console-oauth-config\") on node \"ip-10-0-140-185.ec2.internal\" DevicePath \"\"" Jul 27 17:29:52.935895 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:52.935677 2570 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f1389cc5-47f4-4804-8c4e-836a5181bd76-console-serving-cert\") on node \"ip-10-0-140-185.ec2.internal\" DevicePath \"\"" Jul 27 17:29:53.685616 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:53.685587 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-675f4b885b-bggck" Jul 27 17:29:53.685616 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:53.685591 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-675f4b885b-bggck" event={"ID":"f1389cc5-47f4-4804-8c4e-836a5181bd76","Type":"ContainerDied","Data":"e1b9f25d0ad2e422007431b0062ad40ffca7b829ff63df36abe48ca786b81fc2"} Jul 27 17:29:53.686117 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:53.685667 2570 scope.go:117] "RemoveContainer" containerID="023762b8b11e1512ee5d600c7ac6da17813cb9efc5b5e6c8990ae589913788ba" Jul 27 17:29:53.700649 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:53.700624 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-675f4b885b-bggck"] Jul 27 17:29:53.704691 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:53.704668 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-675f4b885b-bggck"] Jul 27 17:29:55.128349 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:29:55.128317 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1389cc5-47f4-4804-8c4e-836a5181bd76" path="/var/lib/kubelet/pods/f1389cc5-47f4-4804-8c4e-836a5181bd76/volumes" Jul 27 17:30:43.559950 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.559907 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 27 17:30:43.560491 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.560176 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="41e7e826-76ea-4c02-87c8-739b0653b66a" containerName="alertmanager" containerID="cri-o://c3e0f580e654ecba2ebb2afb155e162612da3b7897a5f19620316e86819c0148" gracePeriod=120 Jul 27 17:30:43.560491 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.560255 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="41e7e826-76ea-4c02-87c8-739b0653b66a" containerName="kube-rbac-proxy-web" containerID="cri-o://5af24472f29983b8490f501232ef466f348cc8c6103af90d4c1c5133bcdeb8df" gracePeriod=120 Jul 27 17:30:43.560491 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.560255 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="41e7e826-76ea-4c02-87c8-739b0653b66a" containerName="kube-rbac-proxy-metric" containerID="cri-o://d4a3b4130cbd345d25d484bb2b561d67c83a40fa95d77948cf22d9f401c4c855" gracePeriod=120 Jul 27 17:30:43.560491 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.560302 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="41e7e826-76ea-4c02-87c8-739b0653b66a" containerName="prom-label-proxy" containerID="cri-o://7349ab94536e9a73fc37447f4efa62b7c843f3b0dca3c56b83351744cf69888a" gracePeriod=120 Jul 27 17:30:43.560491 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.560330 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="41e7e826-76ea-4c02-87c8-739b0653b66a" containerName="config-reloader" containerID="cri-o://322d221f68a0492ab95876071778dd8cecb0ab5bde0cdf9f8ce606dfaa2cfc18" gracePeriod=120 Jul 27 17:30:43.560491 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.560340 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="41e7e826-76ea-4c02-87c8-739b0653b66a" containerName="kube-rbac-proxy" containerID="cri-o://b2f4898e0a1243e52057a1a66f0cd0f5cf35858fdb46c56201ffa0323eecb983" gracePeriod=120 Jul 27 17:30:43.806872 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.806846 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:30:43.872236 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.872152 2570 generic.go:356] "Generic (PLEG): container finished" podID="41e7e826-76ea-4c02-87c8-739b0653b66a" containerID="7349ab94536e9a73fc37447f4efa62b7c843f3b0dca3c56b83351744cf69888a" exitCode=0 Jul 27 17:30:43.872236 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.872176 2570 generic.go:356] "Generic (PLEG): container finished" podID="41e7e826-76ea-4c02-87c8-739b0653b66a" containerID="d4a3b4130cbd345d25d484bb2b561d67c83a40fa95d77948cf22d9f401c4c855" exitCode=0 Jul 27 17:30:43.872236 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.872184 2570 generic.go:356] "Generic (PLEG): container finished" podID="41e7e826-76ea-4c02-87c8-739b0653b66a" containerID="b2f4898e0a1243e52057a1a66f0cd0f5cf35858fdb46c56201ffa0323eecb983" exitCode=0 Jul 27 17:30:43.872236 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.872192 2570 generic.go:356] "Generic (PLEG): container finished" podID="41e7e826-76ea-4c02-87c8-739b0653b66a" containerID="5af24472f29983b8490f501232ef466f348cc8c6103af90d4c1c5133bcdeb8df" exitCode=0 Jul 27 17:30:43.872236 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.872199 2570 generic.go:356] "Generic (PLEG): container finished" podID="41e7e826-76ea-4c02-87c8-739b0653b66a" containerID="322d221f68a0492ab95876071778dd8cecb0ab5bde0cdf9f8ce606dfaa2cfc18" exitCode=0 Jul 27 17:30:43.872236 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.872206 2570 generic.go:356] "Generic (PLEG): container finished" podID="41e7e826-76ea-4c02-87c8-739b0653b66a" containerID="c3e0f580e654ecba2ebb2afb155e162612da3b7897a5f19620316e86819c0148" exitCode=0 Jul 27 17:30:43.872236 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.872234 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:30:43.872661 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.872232 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"41e7e826-76ea-4c02-87c8-739b0653b66a","Type":"ContainerDied","Data":"7349ab94536e9a73fc37447f4efa62b7c843f3b0dca3c56b83351744cf69888a"} Jul 27 17:30:43.872661 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.872389 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"41e7e826-76ea-4c02-87c8-739b0653b66a","Type":"ContainerDied","Data":"d4a3b4130cbd345d25d484bb2b561d67c83a40fa95d77948cf22d9f401c4c855"} Jul 27 17:30:43.872661 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.872420 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"41e7e826-76ea-4c02-87c8-739b0653b66a","Type":"ContainerDied","Data":"b2f4898e0a1243e52057a1a66f0cd0f5cf35858fdb46c56201ffa0323eecb983"} Jul 27 17:30:43.872661 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.872447 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"41e7e826-76ea-4c02-87c8-739b0653b66a","Type":"ContainerDied","Data":"5af24472f29983b8490f501232ef466f348cc8c6103af90d4c1c5133bcdeb8df"} Jul 27 17:30:43.872661 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.872460 2570 scope.go:117] "RemoveContainer" containerID="7349ab94536e9a73fc37447f4efa62b7c843f3b0dca3c56b83351744cf69888a" Jul 27 17:30:43.872661 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.872477 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"41e7e826-76ea-4c02-87c8-739b0653b66a","Type":"ContainerDied","Data":"322d221f68a0492ab95876071778dd8cecb0ab5bde0cdf9f8ce606dfaa2cfc18"} Jul 27 17:30:43.872661 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.872503 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"41e7e826-76ea-4c02-87c8-739b0653b66a","Type":"ContainerDied","Data":"c3e0f580e654ecba2ebb2afb155e162612da3b7897a5f19620316e86819c0148"} Jul 27 17:30:43.872661 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.872546 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"41e7e826-76ea-4c02-87c8-739b0653b66a","Type":"ContainerDied","Data":"840eb68207b058db9ef29ff451726d5c11e295b4d0b846e39da35cc53482dc77"} Jul 27 17:30:43.879892 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.879873 2570 scope.go:117] "RemoveContainer" containerID="d4a3b4130cbd345d25d484bb2b561d67c83a40fa95d77948cf22d9f401c4c855" Jul 27 17:30:43.886878 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.886848 2570 scope.go:117] "RemoveContainer" containerID="b2f4898e0a1243e52057a1a66f0cd0f5cf35858fdb46c56201ffa0323eecb983" Jul 27 17:30:43.893212 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.893195 2570 scope.go:117] "RemoveContainer" containerID="5af24472f29983b8490f501232ef466f348cc8c6103af90d4c1c5133bcdeb8df" Jul 27 17:30:43.899635 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.899609 2570 scope.go:117] "RemoveContainer" containerID="322d221f68a0492ab95876071778dd8cecb0ab5bde0cdf9f8ce606dfaa2cfc18" Jul 27 17:30:43.906193 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.906177 2570 scope.go:117] "RemoveContainer" containerID="c3e0f580e654ecba2ebb2afb155e162612da3b7897a5f19620316e86819c0148" Jul 27 17:30:43.914058 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.913886 2570 scope.go:117] "RemoveContainer" containerID="34daf9820eb485de580b293272dfc9ba260df4f93634def819a070074830afe9" Jul 27 17:30:43.921114 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.921097 2570 scope.go:117] "RemoveContainer" containerID="7349ab94536e9a73fc37447f4efa62b7c843f3b0dca3c56b83351744cf69888a" Jul 27 17:30:43.921355 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:30:43.921337 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7349ab94536e9a73fc37447f4efa62b7c843f3b0dca3c56b83351744cf69888a\": container with ID starting with 7349ab94536e9a73fc37447f4efa62b7c843f3b0dca3c56b83351744cf69888a not found: ID does not exist" containerID="7349ab94536e9a73fc37447f4efa62b7c843f3b0dca3c56b83351744cf69888a" Jul 27 17:30:43.921399 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.921365 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7349ab94536e9a73fc37447f4efa62b7c843f3b0dca3c56b83351744cf69888a"} err="failed to get container status \"7349ab94536e9a73fc37447f4efa62b7c843f3b0dca3c56b83351744cf69888a\": rpc error: code = NotFound desc = could not find container \"7349ab94536e9a73fc37447f4efa62b7c843f3b0dca3c56b83351744cf69888a\": container with ID starting with 7349ab94536e9a73fc37447f4efa62b7c843f3b0dca3c56b83351744cf69888a not found: ID does not exist" Jul 27 17:30:43.921399 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.921395 2570 scope.go:117] "RemoveContainer" containerID="d4a3b4130cbd345d25d484bb2b561d67c83a40fa95d77948cf22d9f401c4c855" Jul 27 17:30:43.921625 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:30:43.921608 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4a3b4130cbd345d25d484bb2b561d67c83a40fa95d77948cf22d9f401c4c855\": container with ID starting with d4a3b4130cbd345d25d484bb2b561d67c83a40fa95d77948cf22d9f401c4c855 not found: ID does not exist" containerID="d4a3b4130cbd345d25d484bb2b561d67c83a40fa95d77948cf22d9f401c4c855" Jul 27 17:30:43.921664 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.921630 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4a3b4130cbd345d25d484bb2b561d67c83a40fa95d77948cf22d9f401c4c855"} err="failed to get container status \"d4a3b4130cbd345d25d484bb2b561d67c83a40fa95d77948cf22d9f401c4c855\": rpc error: code = NotFound desc = could not find container \"d4a3b4130cbd345d25d484bb2b561d67c83a40fa95d77948cf22d9f401c4c855\": container with ID starting with d4a3b4130cbd345d25d484bb2b561d67c83a40fa95d77948cf22d9f401c4c855 not found: ID does not exist" Jul 27 17:30:43.921664 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.921644 2570 scope.go:117] "RemoveContainer" containerID="b2f4898e0a1243e52057a1a66f0cd0f5cf35858fdb46c56201ffa0323eecb983" Jul 27 17:30:43.921850 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:30:43.921832 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2f4898e0a1243e52057a1a66f0cd0f5cf35858fdb46c56201ffa0323eecb983\": container with ID starting with b2f4898e0a1243e52057a1a66f0cd0f5cf35858fdb46c56201ffa0323eecb983 not found: ID does not exist" containerID="b2f4898e0a1243e52057a1a66f0cd0f5cf35858fdb46c56201ffa0323eecb983" Jul 27 17:30:43.921921 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.921859 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2f4898e0a1243e52057a1a66f0cd0f5cf35858fdb46c56201ffa0323eecb983"} err="failed to get container status \"b2f4898e0a1243e52057a1a66f0cd0f5cf35858fdb46c56201ffa0323eecb983\": rpc error: code = NotFound desc = could not find container \"b2f4898e0a1243e52057a1a66f0cd0f5cf35858fdb46c56201ffa0323eecb983\": container with ID starting with b2f4898e0a1243e52057a1a66f0cd0f5cf35858fdb46c56201ffa0323eecb983 not found: ID does not exist" Jul 27 17:30:43.921921 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.921883 2570 scope.go:117] "RemoveContainer" containerID="5af24472f29983b8490f501232ef466f348cc8c6103af90d4c1c5133bcdeb8df" Jul 27 17:30:43.922160 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:30:43.922145 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5af24472f29983b8490f501232ef466f348cc8c6103af90d4c1c5133bcdeb8df\": container with ID starting with 5af24472f29983b8490f501232ef466f348cc8c6103af90d4c1c5133bcdeb8df not found: ID does not exist" containerID="5af24472f29983b8490f501232ef466f348cc8c6103af90d4c1c5133bcdeb8df" Jul 27 17:30:43.922197 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.922165 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5af24472f29983b8490f501232ef466f348cc8c6103af90d4c1c5133bcdeb8df"} err="failed to get container status \"5af24472f29983b8490f501232ef466f348cc8c6103af90d4c1c5133bcdeb8df\": rpc error: code = NotFound desc = could not find container \"5af24472f29983b8490f501232ef466f348cc8c6103af90d4c1c5133bcdeb8df\": container with ID starting with 5af24472f29983b8490f501232ef466f348cc8c6103af90d4c1c5133bcdeb8df not found: ID does not exist" Jul 27 17:30:43.922197 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.922178 2570 scope.go:117] "RemoveContainer" containerID="322d221f68a0492ab95876071778dd8cecb0ab5bde0cdf9f8ce606dfaa2cfc18" Jul 27 17:30:43.922371 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:30:43.922353 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"322d221f68a0492ab95876071778dd8cecb0ab5bde0cdf9f8ce606dfaa2cfc18\": container with ID starting with 322d221f68a0492ab95876071778dd8cecb0ab5bde0cdf9f8ce606dfaa2cfc18 not found: ID does not exist" containerID="322d221f68a0492ab95876071778dd8cecb0ab5bde0cdf9f8ce606dfaa2cfc18" Jul 27 17:30:43.922406 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.922377 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"322d221f68a0492ab95876071778dd8cecb0ab5bde0cdf9f8ce606dfaa2cfc18"} err="failed to get container status \"322d221f68a0492ab95876071778dd8cecb0ab5bde0cdf9f8ce606dfaa2cfc18\": rpc error: code = NotFound desc = could not find container \"322d221f68a0492ab95876071778dd8cecb0ab5bde0cdf9f8ce606dfaa2cfc18\": container with ID starting with 322d221f68a0492ab95876071778dd8cecb0ab5bde0cdf9f8ce606dfaa2cfc18 not found: ID does not exist" Jul 27 17:30:43.922406 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.922393 2570 scope.go:117] "RemoveContainer" containerID="c3e0f580e654ecba2ebb2afb155e162612da3b7897a5f19620316e86819c0148" Jul 27 17:30:43.922616 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:30:43.922600 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3e0f580e654ecba2ebb2afb155e162612da3b7897a5f19620316e86819c0148\": container with ID starting with c3e0f580e654ecba2ebb2afb155e162612da3b7897a5f19620316e86819c0148 not found: ID does not exist" containerID="c3e0f580e654ecba2ebb2afb155e162612da3b7897a5f19620316e86819c0148" Jul 27 17:30:43.922664 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.922620 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3e0f580e654ecba2ebb2afb155e162612da3b7897a5f19620316e86819c0148"} err="failed to get container status \"c3e0f580e654ecba2ebb2afb155e162612da3b7897a5f19620316e86819c0148\": rpc error: code = NotFound desc = could not find container \"c3e0f580e654ecba2ebb2afb155e162612da3b7897a5f19620316e86819c0148\": container with ID starting with c3e0f580e654ecba2ebb2afb155e162612da3b7897a5f19620316e86819c0148 not found: ID does not exist" Jul 27 17:30:43.922664 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.922632 2570 scope.go:117] "RemoveContainer" containerID="34daf9820eb485de580b293272dfc9ba260df4f93634def819a070074830afe9" Jul 27 17:30:43.922806 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:30:43.922791 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34daf9820eb485de580b293272dfc9ba260df4f93634def819a070074830afe9\": container with ID starting with 34daf9820eb485de580b293272dfc9ba260df4f93634def819a070074830afe9 not found: ID does not exist" containerID="34daf9820eb485de580b293272dfc9ba260df4f93634def819a070074830afe9" Jul 27 17:30:43.922851 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.922809 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34daf9820eb485de580b293272dfc9ba260df4f93634def819a070074830afe9"} err="failed to get container status \"34daf9820eb485de580b293272dfc9ba260df4f93634def819a070074830afe9\": rpc error: code = NotFound desc = could not find container \"34daf9820eb485de580b293272dfc9ba260df4f93634def819a070074830afe9\": container with ID starting with 34daf9820eb485de580b293272dfc9ba260df4f93634def819a070074830afe9 not found: ID does not exist" Jul 27 17:30:43.922851 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.922829 2570 scope.go:117] "RemoveContainer" containerID="7349ab94536e9a73fc37447f4efa62b7c843f3b0dca3c56b83351744cf69888a" Jul 27 17:30:43.923001 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.922987 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7349ab94536e9a73fc37447f4efa62b7c843f3b0dca3c56b83351744cf69888a"} err="failed to get container status \"7349ab94536e9a73fc37447f4efa62b7c843f3b0dca3c56b83351744cf69888a\": rpc error: code = NotFound desc = could not find container \"7349ab94536e9a73fc37447f4efa62b7c843f3b0dca3c56b83351744cf69888a\": container with ID starting with 7349ab94536e9a73fc37447f4efa62b7c843f3b0dca3c56b83351744cf69888a not found: ID does not exist" Jul 27 17:30:43.923046 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.923001 2570 scope.go:117] "RemoveContainer" containerID="d4a3b4130cbd345d25d484bb2b561d67c83a40fa95d77948cf22d9f401c4c855" Jul 27 17:30:43.923230 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.923214 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4a3b4130cbd345d25d484bb2b561d67c83a40fa95d77948cf22d9f401c4c855"} err="failed to get container status \"d4a3b4130cbd345d25d484bb2b561d67c83a40fa95d77948cf22d9f401c4c855\": rpc error: code = NotFound desc = could not find container \"d4a3b4130cbd345d25d484bb2b561d67c83a40fa95d77948cf22d9f401c4c855\": container with ID starting with d4a3b4130cbd345d25d484bb2b561d67c83a40fa95d77948cf22d9f401c4c855 not found: ID does not exist" Jul 27 17:30:43.923281 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.923232 2570 scope.go:117] "RemoveContainer" containerID="b2f4898e0a1243e52057a1a66f0cd0f5cf35858fdb46c56201ffa0323eecb983" Jul 27 17:30:43.923436 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.923418 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2f4898e0a1243e52057a1a66f0cd0f5cf35858fdb46c56201ffa0323eecb983"} err="failed to get container status \"b2f4898e0a1243e52057a1a66f0cd0f5cf35858fdb46c56201ffa0323eecb983\": rpc error: code = NotFound desc = could not find container \"b2f4898e0a1243e52057a1a66f0cd0f5cf35858fdb46c56201ffa0323eecb983\": container with ID starting with b2f4898e0a1243e52057a1a66f0cd0f5cf35858fdb46c56201ffa0323eecb983 not found: ID does not exist" Jul 27 17:30:43.923474 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.923436 2570 scope.go:117] "RemoveContainer" containerID="5af24472f29983b8490f501232ef466f348cc8c6103af90d4c1c5133bcdeb8df" Jul 27 17:30:43.923685 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.923664 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5af24472f29983b8490f501232ef466f348cc8c6103af90d4c1c5133bcdeb8df"} err="failed to get container status \"5af24472f29983b8490f501232ef466f348cc8c6103af90d4c1c5133bcdeb8df\": rpc error: code = NotFound desc = could not find container \"5af24472f29983b8490f501232ef466f348cc8c6103af90d4c1c5133bcdeb8df\": container with ID starting with 5af24472f29983b8490f501232ef466f348cc8c6103af90d4c1c5133bcdeb8df not found: ID does not exist" Jul 27 17:30:43.923739 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.923686 2570 scope.go:117] "RemoveContainer" containerID="322d221f68a0492ab95876071778dd8cecb0ab5bde0cdf9f8ce606dfaa2cfc18" Jul 27 17:30:43.923894 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.923872 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"322d221f68a0492ab95876071778dd8cecb0ab5bde0cdf9f8ce606dfaa2cfc18"} err="failed to get container status \"322d221f68a0492ab95876071778dd8cecb0ab5bde0cdf9f8ce606dfaa2cfc18\": rpc error: code = NotFound desc = could not find container \"322d221f68a0492ab95876071778dd8cecb0ab5bde0cdf9f8ce606dfaa2cfc18\": container with ID starting with 322d221f68a0492ab95876071778dd8cecb0ab5bde0cdf9f8ce606dfaa2cfc18 not found: ID does not exist" Jul 27 17:30:43.923960 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.923895 2570 scope.go:117] "RemoveContainer" containerID="c3e0f580e654ecba2ebb2afb155e162612da3b7897a5f19620316e86819c0148" Jul 27 17:30:43.924127 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.924110 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3e0f580e654ecba2ebb2afb155e162612da3b7897a5f19620316e86819c0148"} err="failed to get container status \"c3e0f580e654ecba2ebb2afb155e162612da3b7897a5f19620316e86819c0148\": rpc error: code = NotFound desc = could not find container \"c3e0f580e654ecba2ebb2afb155e162612da3b7897a5f19620316e86819c0148\": container with ID starting with c3e0f580e654ecba2ebb2afb155e162612da3b7897a5f19620316e86819c0148 not found: ID does not exist" Jul 27 17:30:43.924178 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.924128 2570 scope.go:117] "RemoveContainer" containerID="34daf9820eb485de580b293272dfc9ba260df4f93634def819a070074830afe9" Jul 27 17:30:43.924322 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.924306 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34daf9820eb485de580b293272dfc9ba260df4f93634def819a070074830afe9"} err="failed to get container status \"34daf9820eb485de580b293272dfc9ba260df4f93634def819a070074830afe9\": rpc error: code = NotFound desc = could not find container \"34daf9820eb485de580b293272dfc9ba260df4f93634def819a070074830afe9\": container with ID starting with 34daf9820eb485de580b293272dfc9ba260df4f93634def819a070074830afe9 not found: ID does not exist" Jul 27 17:30:43.924367 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.924323 2570 scope.go:117] "RemoveContainer" containerID="7349ab94536e9a73fc37447f4efa62b7c843f3b0dca3c56b83351744cf69888a" Jul 27 17:30:43.924551 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.924515 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7349ab94536e9a73fc37447f4efa62b7c843f3b0dca3c56b83351744cf69888a"} err="failed to get container status \"7349ab94536e9a73fc37447f4efa62b7c843f3b0dca3c56b83351744cf69888a\": rpc error: code = NotFound desc = could not find container \"7349ab94536e9a73fc37447f4efa62b7c843f3b0dca3c56b83351744cf69888a\": container with ID starting with 7349ab94536e9a73fc37447f4efa62b7c843f3b0dca3c56b83351744cf69888a not found: ID does not exist" Jul 27 17:30:43.924551 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.924549 2570 scope.go:117] "RemoveContainer" containerID="d4a3b4130cbd345d25d484bb2b561d67c83a40fa95d77948cf22d9f401c4c855" Jul 27 17:30:43.924773 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.924757 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4a3b4130cbd345d25d484bb2b561d67c83a40fa95d77948cf22d9f401c4c855"} err="failed to get container status \"d4a3b4130cbd345d25d484bb2b561d67c83a40fa95d77948cf22d9f401c4c855\": rpc error: code = NotFound desc = could not find container \"d4a3b4130cbd345d25d484bb2b561d67c83a40fa95d77948cf22d9f401c4c855\": container with ID starting with d4a3b4130cbd345d25d484bb2b561d67c83a40fa95d77948cf22d9f401c4c855 not found: ID does not exist" Jul 27 17:30:43.924773 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.924773 2570 scope.go:117] "RemoveContainer" containerID="b2f4898e0a1243e52057a1a66f0cd0f5cf35858fdb46c56201ffa0323eecb983" Jul 27 17:30:43.924969 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.924953 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2f4898e0a1243e52057a1a66f0cd0f5cf35858fdb46c56201ffa0323eecb983"} err="failed to get container status \"b2f4898e0a1243e52057a1a66f0cd0f5cf35858fdb46c56201ffa0323eecb983\": rpc error: code = NotFound desc = could not find container \"b2f4898e0a1243e52057a1a66f0cd0f5cf35858fdb46c56201ffa0323eecb983\": container with ID starting with b2f4898e0a1243e52057a1a66f0cd0f5cf35858fdb46c56201ffa0323eecb983 not found: ID does not exist" Jul 27 17:30:43.925019 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.924970 2570 scope.go:117] "RemoveContainer" containerID="5af24472f29983b8490f501232ef466f348cc8c6103af90d4c1c5133bcdeb8df" Jul 27 17:30:43.925167 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.925146 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5af24472f29983b8490f501232ef466f348cc8c6103af90d4c1c5133bcdeb8df"} err="failed to get container status \"5af24472f29983b8490f501232ef466f348cc8c6103af90d4c1c5133bcdeb8df\": rpc error: code = NotFound desc = could not find container \"5af24472f29983b8490f501232ef466f348cc8c6103af90d4c1c5133bcdeb8df\": container with ID starting with 5af24472f29983b8490f501232ef466f348cc8c6103af90d4c1c5133bcdeb8df not found: ID does not exist" Jul 27 17:30:43.925167 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.925167 2570 scope.go:117] "RemoveContainer" containerID="322d221f68a0492ab95876071778dd8cecb0ab5bde0cdf9f8ce606dfaa2cfc18" Jul 27 17:30:43.925379 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.925363 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"322d221f68a0492ab95876071778dd8cecb0ab5bde0cdf9f8ce606dfaa2cfc18"} err="failed to get container status \"322d221f68a0492ab95876071778dd8cecb0ab5bde0cdf9f8ce606dfaa2cfc18\": rpc error: code = NotFound desc = could not find container \"322d221f68a0492ab95876071778dd8cecb0ab5bde0cdf9f8ce606dfaa2cfc18\": container with ID starting with 322d221f68a0492ab95876071778dd8cecb0ab5bde0cdf9f8ce606dfaa2cfc18 not found: ID does not exist" Jul 27 17:30:43.925379 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.925378 2570 scope.go:117] "RemoveContainer" containerID="c3e0f580e654ecba2ebb2afb155e162612da3b7897a5f19620316e86819c0148" Jul 27 17:30:43.925604 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.925583 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3e0f580e654ecba2ebb2afb155e162612da3b7897a5f19620316e86819c0148"} err="failed to get container status \"c3e0f580e654ecba2ebb2afb155e162612da3b7897a5f19620316e86819c0148\": rpc error: code = NotFound desc = could not find container \"c3e0f580e654ecba2ebb2afb155e162612da3b7897a5f19620316e86819c0148\": container with ID starting with c3e0f580e654ecba2ebb2afb155e162612da3b7897a5f19620316e86819c0148 not found: ID does not exist" Jul 27 17:30:43.925673 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.925611 2570 scope.go:117] "RemoveContainer" containerID="34daf9820eb485de580b293272dfc9ba260df4f93634def819a070074830afe9" Jul 27 17:30:43.925817 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.925800 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34daf9820eb485de580b293272dfc9ba260df4f93634def819a070074830afe9"} err="failed to get container status \"34daf9820eb485de580b293272dfc9ba260df4f93634def819a070074830afe9\": rpc error: code = NotFound desc = could not find container \"34daf9820eb485de580b293272dfc9ba260df4f93634def819a070074830afe9\": container with ID starting with 34daf9820eb485de580b293272dfc9ba260df4f93634def819a070074830afe9 not found: ID does not exist" Jul 27 17:30:43.925857 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.925819 2570 scope.go:117] "RemoveContainer" containerID="7349ab94536e9a73fc37447f4efa62b7c843f3b0dca3c56b83351744cf69888a" Jul 27 17:30:43.926020 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.926004 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7349ab94536e9a73fc37447f4efa62b7c843f3b0dca3c56b83351744cf69888a"} err="failed to get container status \"7349ab94536e9a73fc37447f4efa62b7c843f3b0dca3c56b83351744cf69888a\": rpc error: code = NotFound desc = could not find container \"7349ab94536e9a73fc37447f4efa62b7c843f3b0dca3c56b83351744cf69888a\": container with ID starting with 7349ab94536e9a73fc37447f4efa62b7c843f3b0dca3c56b83351744cf69888a not found: ID does not exist" Jul 27 17:30:43.926067 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.926023 2570 scope.go:117] "RemoveContainer" containerID="d4a3b4130cbd345d25d484bb2b561d67c83a40fa95d77948cf22d9f401c4c855" Jul 27 17:30:43.926210 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.926194 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4a3b4130cbd345d25d484bb2b561d67c83a40fa95d77948cf22d9f401c4c855"} err="failed to get container status \"d4a3b4130cbd345d25d484bb2b561d67c83a40fa95d77948cf22d9f401c4c855\": rpc error: code = NotFound desc = could not find container \"d4a3b4130cbd345d25d484bb2b561d67c83a40fa95d77948cf22d9f401c4c855\": container with ID starting with d4a3b4130cbd345d25d484bb2b561d67c83a40fa95d77948cf22d9f401c4c855 not found: ID does not exist" Jul 27 17:30:43.926254 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.926210 2570 scope.go:117] "RemoveContainer" containerID="b2f4898e0a1243e52057a1a66f0cd0f5cf35858fdb46c56201ffa0323eecb983" Jul 27 17:30:43.926410 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.926394 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2f4898e0a1243e52057a1a66f0cd0f5cf35858fdb46c56201ffa0323eecb983"} err="failed to get container status \"b2f4898e0a1243e52057a1a66f0cd0f5cf35858fdb46c56201ffa0323eecb983\": rpc error: code = NotFound desc = could not find container \"b2f4898e0a1243e52057a1a66f0cd0f5cf35858fdb46c56201ffa0323eecb983\": container with ID starting with b2f4898e0a1243e52057a1a66f0cd0f5cf35858fdb46c56201ffa0323eecb983 not found: ID does not exist" Jul 27 17:30:43.926462 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.926410 2570 scope.go:117] "RemoveContainer" containerID="5af24472f29983b8490f501232ef466f348cc8c6103af90d4c1c5133bcdeb8df" Jul 27 17:30:43.926624 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.926603 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5af24472f29983b8490f501232ef466f348cc8c6103af90d4c1c5133bcdeb8df"} err="failed to get container status \"5af24472f29983b8490f501232ef466f348cc8c6103af90d4c1c5133bcdeb8df\": rpc error: code = NotFound desc = could not find container \"5af24472f29983b8490f501232ef466f348cc8c6103af90d4c1c5133bcdeb8df\": container with ID starting with 5af24472f29983b8490f501232ef466f348cc8c6103af90d4c1c5133bcdeb8df not found: ID does not exist" Jul 27 17:30:43.926675 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.926625 2570 scope.go:117] "RemoveContainer" containerID="322d221f68a0492ab95876071778dd8cecb0ab5bde0cdf9f8ce606dfaa2cfc18" Jul 27 17:30:43.926803 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.926789 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"322d221f68a0492ab95876071778dd8cecb0ab5bde0cdf9f8ce606dfaa2cfc18"} err="failed to get container status \"322d221f68a0492ab95876071778dd8cecb0ab5bde0cdf9f8ce606dfaa2cfc18\": rpc error: code = NotFound desc = could not find container \"322d221f68a0492ab95876071778dd8cecb0ab5bde0cdf9f8ce606dfaa2cfc18\": container with ID starting with 322d221f68a0492ab95876071778dd8cecb0ab5bde0cdf9f8ce606dfaa2cfc18 not found: ID does not exist" Jul 27 17:30:43.926845 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.926803 2570 scope.go:117] "RemoveContainer" containerID="c3e0f580e654ecba2ebb2afb155e162612da3b7897a5f19620316e86819c0148" Jul 27 17:30:43.926967 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.926952 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3e0f580e654ecba2ebb2afb155e162612da3b7897a5f19620316e86819c0148"} err="failed to get container status \"c3e0f580e654ecba2ebb2afb155e162612da3b7897a5f19620316e86819c0148\": rpc error: code = NotFound desc = could not find container \"c3e0f580e654ecba2ebb2afb155e162612da3b7897a5f19620316e86819c0148\": container with ID starting with c3e0f580e654ecba2ebb2afb155e162612da3b7897a5f19620316e86819c0148 not found: ID does not exist" Jul 27 17:30:43.927018 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.926966 2570 scope.go:117] "RemoveContainer" containerID="34daf9820eb485de580b293272dfc9ba260df4f93634def819a070074830afe9" Jul 27 17:30:43.927173 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.927157 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34daf9820eb485de580b293272dfc9ba260df4f93634def819a070074830afe9"} err="failed to get container status \"34daf9820eb485de580b293272dfc9ba260df4f93634def819a070074830afe9\": rpc error: code = NotFound desc = could not find container \"34daf9820eb485de580b293272dfc9ba260df4f93634def819a070074830afe9\": container with ID starting with 34daf9820eb485de580b293272dfc9ba260df4f93634def819a070074830afe9 not found: ID does not exist" Jul 27 17:30:43.927236 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.927172 2570 scope.go:117] "RemoveContainer" containerID="7349ab94536e9a73fc37447f4efa62b7c843f3b0dca3c56b83351744cf69888a" Jul 27 17:30:43.927376 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.927360 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7349ab94536e9a73fc37447f4efa62b7c843f3b0dca3c56b83351744cf69888a"} err="failed to get container status \"7349ab94536e9a73fc37447f4efa62b7c843f3b0dca3c56b83351744cf69888a\": rpc error: code = NotFound desc = could not find container \"7349ab94536e9a73fc37447f4efa62b7c843f3b0dca3c56b83351744cf69888a\": container with ID starting with 7349ab94536e9a73fc37447f4efa62b7c843f3b0dca3c56b83351744cf69888a not found: ID does not exist" Jul 27 17:30:43.927423 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.927376 2570 scope.go:117] "RemoveContainer" containerID="d4a3b4130cbd345d25d484bb2b561d67c83a40fa95d77948cf22d9f401c4c855" Jul 27 17:30:43.927595 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.927580 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4a3b4130cbd345d25d484bb2b561d67c83a40fa95d77948cf22d9f401c4c855"} err="failed to get container status \"d4a3b4130cbd345d25d484bb2b561d67c83a40fa95d77948cf22d9f401c4c855\": rpc error: code = NotFound desc = could not find container \"d4a3b4130cbd345d25d484bb2b561d67c83a40fa95d77948cf22d9f401c4c855\": container with ID starting with d4a3b4130cbd345d25d484bb2b561d67c83a40fa95d77948cf22d9f401c4c855 not found: ID does not exist" Jul 27 17:30:43.927595 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.927595 2570 scope.go:117] "RemoveContainer" containerID="b2f4898e0a1243e52057a1a66f0cd0f5cf35858fdb46c56201ffa0323eecb983" Jul 27 17:30:43.927793 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.927777 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2f4898e0a1243e52057a1a66f0cd0f5cf35858fdb46c56201ffa0323eecb983"} err="failed to get container status \"b2f4898e0a1243e52057a1a66f0cd0f5cf35858fdb46c56201ffa0323eecb983\": rpc error: code = NotFound desc = could not find container \"b2f4898e0a1243e52057a1a66f0cd0f5cf35858fdb46c56201ffa0323eecb983\": container with ID starting with b2f4898e0a1243e52057a1a66f0cd0f5cf35858fdb46c56201ffa0323eecb983 not found: ID does not exist" Jul 27 17:30:43.927836 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.927793 2570 scope.go:117] "RemoveContainer" containerID="5af24472f29983b8490f501232ef466f348cc8c6103af90d4c1c5133bcdeb8df" Jul 27 17:30:43.927986 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.927970 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5af24472f29983b8490f501232ef466f348cc8c6103af90d4c1c5133bcdeb8df"} err="failed to get container status \"5af24472f29983b8490f501232ef466f348cc8c6103af90d4c1c5133bcdeb8df\": rpc error: code = NotFound desc = could not find container \"5af24472f29983b8490f501232ef466f348cc8c6103af90d4c1c5133bcdeb8df\": container with ID starting with 5af24472f29983b8490f501232ef466f348cc8c6103af90d4c1c5133bcdeb8df not found: ID does not exist" Jul 27 17:30:43.927986 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.927985 2570 scope.go:117] "RemoveContainer" containerID="322d221f68a0492ab95876071778dd8cecb0ab5bde0cdf9f8ce606dfaa2cfc18" Jul 27 17:30:43.928172 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.928158 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"322d221f68a0492ab95876071778dd8cecb0ab5bde0cdf9f8ce606dfaa2cfc18"} err="failed to get container status \"322d221f68a0492ab95876071778dd8cecb0ab5bde0cdf9f8ce606dfaa2cfc18\": rpc error: code = NotFound desc = could not find container \"322d221f68a0492ab95876071778dd8cecb0ab5bde0cdf9f8ce606dfaa2cfc18\": container with ID starting with 322d221f68a0492ab95876071778dd8cecb0ab5bde0cdf9f8ce606dfaa2cfc18 not found: ID does not exist" Jul 27 17:30:43.928219 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.928173 2570 scope.go:117] "RemoveContainer" containerID="c3e0f580e654ecba2ebb2afb155e162612da3b7897a5f19620316e86819c0148" Jul 27 17:30:43.928373 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.928359 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3e0f580e654ecba2ebb2afb155e162612da3b7897a5f19620316e86819c0148"} err="failed to get container status \"c3e0f580e654ecba2ebb2afb155e162612da3b7897a5f19620316e86819c0148\": rpc error: code = NotFound desc = could not find container \"c3e0f580e654ecba2ebb2afb155e162612da3b7897a5f19620316e86819c0148\": container with ID starting with c3e0f580e654ecba2ebb2afb155e162612da3b7897a5f19620316e86819c0148 not found: ID does not exist" Jul 27 17:30:43.928417 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.928373 2570 scope.go:117] "RemoveContainer" containerID="34daf9820eb485de580b293272dfc9ba260df4f93634def819a070074830afe9" Jul 27 17:30:43.928593 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.928576 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34daf9820eb485de580b293272dfc9ba260df4f93634def819a070074830afe9"} err="failed to get container status \"34daf9820eb485de580b293272dfc9ba260df4f93634def819a070074830afe9\": rpc error: code = NotFound desc = could not find container \"34daf9820eb485de580b293272dfc9ba260df4f93634def819a070074830afe9\": container with ID starting with 34daf9820eb485de580b293272dfc9ba260df4f93634def819a070074830afe9 not found: ID does not exist" Jul 27 17:30:43.928593 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.928592 2570 scope.go:117] "RemoveContainer" containerID="7349ab94536e9a73fc37447f4efa62b7c843f3b0dca3c56b83351744cf69888a" Jul 27 17:30:43.928781 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.928764 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7349ab94536e9a73fc37447f4efa62b7c843f3b0dca3c56b83351744cf69888a"} err="failed to get container status \"7349ab94536e9a73fc37447f4efa62b7c843f3b0dca3c56b83351744cf69888a\": rpc error: code = NotFound desc = could not find container \"7349ab94536e9a73fc37447f4efa62b7c843f3b0dca3c56b83351744cf69888a\": container with ID starting with 7349ab94536e9a73fc37447f4efa62b7c843f3b0dca3c56b83351744cf69888a not found: ID does not exist" Jul 27 17:30:43.928824 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.928781 2570 scope.go:117] "RemoveContainer" containerID="d4a3b4130cbd345d25d484bb2b561d67c83a40fa95d77948cf22d9f401c4c855" Jul 27 17:30:43.928993 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.928977 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4a3b4130cbd345d25d484bb2b561d67c83a40fa95d77948cf22d9f401c4c855"} err="failed to get container status \"d4a3b4130cbd345d25d484bb2b561d67c83a40fa95d77948cf22d9f401c4c855\": rpc error: code = NotFound desc = could not find container \"d4a3b4130cbd345d25d484bb2b561d67c83a40fa95d77948cf22d9f401c4c855\": container with ID starting with d4a3b4130cbd345d25d484bb2b561d67c83a40fa95d77948cf22d9f401c4c855 not found: ID does not exist" Jul 27 17:30:43.928993 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.928993 2570 scope.go:117] "RemoveContainer" containerID="b2f4898e0a1243e52057a1a66f0cd0f5cf35858fdb46c56201ffa0323eecb983" Jul 27 17:30:43.929184 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.929169 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2f4898e0a1243e52057a1a66f0cd0f5cf35858fdb46c56201ffa0323eecb983"} err="failed to get container status \"b2f4898e0a1243e52057a1a66f0cd0f5cf35858fdb46c56201ffa0323eecb983\": rpc error: code = NotFound desc = could not find container \"b2f4898e0a1243e52057a1a66f0cd0f5cf35858fdb46c56201ffa0323eecb983\": container with ID starting with b2f4898e0a1243e52057a1a66f0cd0f5cf35858fdb46c56201ffa0323eecb983 not found: ID does not exist" Jul 27 17:30:43.929225 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.929184 2570 scope.go:117] "RemoveContainer" containerID="5af24472f29983b8490f501232ef466f348cc8c6103af90d4c1c5133bcdeb8df" Jul 27 17:30:43.929404 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.929386 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5af24472f29983b8490f501232ef466f348cc8c6103af90d4c1c5133bcdeb8df"} err="failed to get container status \"5af24472f29983b8490f501232ef466f348cc8c6103af90d4c1c5133bcdeb8df\": rpc error: code = NotFound desc = could not find container \"5af24472f29983b8490f501232ef466f348cc8c6103af90d4c1c5133bcdeb8df\": container with ID starting with 5af24472f29983b8490f501232ef466f348cc8c6103af90d4c1c5133bcdeb8df not found: ID does not exist" Jul 27 17:30:43.929456 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.929403 2570 scope.go:117] "RemoveContainer" containerID="322d221f68a0492ab95876071778dd8cecb0ab5bde0cdf9f8ce606dfaa2cfc18" Jul 27 17:30:43.929634 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.929616 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"322d221f68a0492ab95876071778dd8cecb0ab5bde0cdf9f8ce606dfaa2cfc18"} err="failed to get container status \"322d221f68a0492ab95876071778dd8cecb0ab5bde0cdf9f8ce606dfaa2cfc18\": rpc error: code = NotFound desc = could not find container \"322d221f68a0492ab95876071778dd8cecb0ab5bde0cdf9f8ce606dfaa2cfc18\": container with ID starting with 322d221f68a0492ab95876071778dd8cecb0ab5bde0cdf9f8ce606dfaa2cfc18 not found: ID does not exist" Jul 27 17:30:43.929687 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.929634 2570 scope.go:117] "RemoveContainer" containerID="c3e0f580e654ecba2ebb2afb155e162612da3b7897a5f19620316e86819c0148" Jul 27 17:30:43.929825 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.929804 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3e0f580e654ecba2ebb2afb155e162612da3b7897a5f19620316e86819c0148"} err="failed to get container status \"c3e0f580e654ecba2ebb2afb155e162612da3b7897a5f19620316e86819c0148\": rpc error: code = NotFound desc = could not find container \"c3e0f580e654ecba2ebb2afb155e162612da3b7897a5f19620316e86819c0148\": container with ID starting with c3e0f580e654ecba2ebb2afb155e162612da3b7897a5f19620316e86819c0148 not found: ID does not exist" Jul 27 17:30:43.929866 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.929825 2570 scope.go:117] "RemoveContainer" containerID="34daf9820eb485de580b293272dfc9ba260df4f93634def819a070074830afe9" Jul 27 17:30:43.930038 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.930020 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34daf9820eb485de580b293272dfc9ba260df4f93634def819a070074830afe9"} err="failed to get container status \"34daf9820eb485de580b293272dfc9ba260df4f93634def819a070074830afe9\": rpc error: code = NotFound desc = could not find container \"34daf9820eb485de580b293272dfc9ba260df4f93634def819a070074830afe9\": container with ID starting with 34daf9820eb485de580b293272dfc9ba260df4f93634def819a070074830afe9 not found: ID does not exist" Jul 27 17:30:43.958343 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.958309 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/41e7e826-76ea-4c02-87c8-739b0653b66a-config-volume\") pod \"41e7e826-76ea-4c02-87c8-739b0653b66a\" (UID: \"41e7e826-76ea-4c02-87c8-739b0653b66a\") " Jul 27 17:30:43.958453 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.958370 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5qvbj\" (UniqueName: \"kubernetes.io/projected/41e7e826-76ea-4c02-87c8-739b0653b66a-kube-api-access-5qvbj\") pod \"41e7e826-76ea-4c02-87c8-739b0653b66a\" (UID: \"41e7e826-76ea-4c02-87c8-739b0653b66a\") " Jul 27 17:30:43.958453 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.958391 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/41e7e826-76ea-4c02-87c8-739b0653b66a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"41e7e826-76ea-4c02-87c8-739b0653b66a\" (UID: \"41e7e826-76ea-4c02-87c8-739b0653b66a\") " Jul 27 17:30:43.958453 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.958416 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/41e7e826-76ea-4c02-87c8-739b0653b66a-alertmanager-main-db\") pod \"41e7e826-76ea-4c02-87c8-739b0653b66a\" (UID: \"41e7e826-76ea-4c02-87c8-739b0653b66a\") " Jul 27 17:30:43.958623 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.958473 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/41e7e826-76ea-4c02-87c8-739b0653b66a-secret-alertmanager-kube-rbac-proxy\") pod \"41e7e826-76ea-4c02-87c8-739b0653b66a\" (UID: \"41e7e826-76ea-4c02-87c8-739b0653b66a\") " Jul 27 17:30:43.958623 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.958499 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/41e7e826-76ea-4c02-87c8-739b0653b66a-tls-assets\") pod \"41e7e826-76ea-4c02-87c8-739b0653b66a\" (UID: \"41e7e826-76ea-4c02-87c8-739b0653b66a\") " Jul 27 17:30:43.958623 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.958557 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/41e7e826-76ea-4c02-87c8-739b0653b66a-secret-alertmanager-kube-rbac-proxy-web\") pod \"41e7e826-76ea-4c02-87c8-739b0653b66a\" (UID: \"41e7e826-76ea-4c02-87c8-739b0653b66a\") " Jul 27 17:30:43.958623 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.958595 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/41e7e826-76ea-4c02-87c8-739b0653b66a-web-config\") pod \"41e7e826-76ea-4c02-87c8-739b0653b66a\" (UID: \"41e7e826-76ea-4c02-87c8-739b0653b66a\") " Jul 27 17:30:43.958623 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.958620 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41e7e826-76ea-4c02-87c8-739b0653b66a-alertmanager-trusted-ca-bundle\") pod \"41e7e826-76ea-4c02-87c8-739b0653b66a\" (UID: \"41e7e826-76ea-4c02-87c8-739b0653b66a\") " Jul 27 17:30:43.958872 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.958652 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/41e7e826-76ea-4c02-87c8-739b0653b66a-config-out\") pod \"41e7e826-76ea-4c02-87c8-739b0653b66a\" (UID: \"41e7e826-76ea-4c02-87c8-739b0653b66a\") " Jul 27 17:30:43.958872 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.958679 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41e7e826-76ea-4c02-87c8-739b0653b66a-metrics-client-ca\") pod \"41e7e826-76ea-4c02-87c8-739b0653b66a\" (UID: \"41e7e826-76ea-4c02-87c8-739b0653b66a\") " Jul 27 17:30:43.958872 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.958708 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/41e7e826-76ea-4c02-87c8-739b0653b66a-cluster-tls-config\") pod \"41e7e826-76ea-4c02-87c8-739b0653b66a\" (UID: \"41e7e826-76ea-4c02-87c8-739b0653b66a\") " Jul 27 17:30:43.958872 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.958741 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/41e7e826-76ea-4c02-87c8-739b0653b66a-secret-alertmanager-main-tls\") pod \"41e7e826-76ea-4c02-87c8-739b0653b66a\" (UID: \"41e7e826-76ea-4c02-87c8-739b0653b66a\") " Jul 27 17:30:43.958872 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.958809 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41e7e826-76ea-4c02-87c8-739b0653b66a-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "41e7e826-76ea-4c02-87c8-739b0653b66a" (UID: "41e7e826-76ea-4c02-87c8-739b0653b66a"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:30:43.959125 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.959034 2570 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/41e7e826-76ea-4c02-87c8-739b0653b66a-alertmanager-main-db\") on node \"ip-10-0-140-185.ec2.internal\" DevicePath \"\"" Jul 27 17:30:43.959935 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.959904 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41e7e826-76ea-4c02-87c8-739b0653b66a-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "41e7e826-76ea-4c02-87c8-739b0653b66a" (UID: "41e7e826-76ea-4c02-87c8-739b0653b66a"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:30:43.960058 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.959911 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41e7e826-76ea-4c02-87c8-739b0653b66a-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "41e7e826-76ea-4c02-87c8-739b0653b66a" (UID: "41e7e826-76ea-4c02-87c8-739b0653b66a"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:30:43.961345 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.961304 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41e7e826-76ea-4c02-87c8-739b0653b66a-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "41e7e826-76ea-4c02-87c8-739b0653b66a" (UID: "41e7e826-76ea-4c02-87c8-739b0653b66a"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 17:30:43.961812 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.961783 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41e7e826-76ea-4c02-87c8-739b0653b66a-config-out" (OuterVolumeSpecName: "config-out") pod "41e7e826-76ea-4c02-87c8-739b0653b66a" (UID: "41e7e826-76ea-4c02-87c8-739b0653b66a"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 17:30:43.961812 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.961798 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41e7e826-76ea-4c02-87c8-739b0653b66a-kube-api-access-5qvbj" (OuterVolumeSpecName: "kube-api-access-5qvbj") pod "41e7e826-76ea-4c02-87c8-739b0653b66a" (UID: "41e7e826-76ea-4c02-87c8-739b0653b66a"). InnerVolumeSpecName "kube-api-access-5qvbj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:30:43.961957 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.961830 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41e7e826-76ea-4c02-87c8-739b0653b66a-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "41e7e826-76ea-4c02-87c8-739b0653b66a" (UID: "41e7e826-76ea-4c02-87c8-739b0653b66a"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 17:30:43.962071 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.962031 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41e7e826-76ea-4c02-87c8-739b0653b66a-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "41e7e826-76ea-4c02-87c8-739b0653b66a" (UID: "41e7e826-76ea-4c02-87c8-739b0653b66a"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 17:30:43.962153 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.962125 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41e7e826-76ea-4c02-87c8-739b0653b66a-config-volume" (OuterVolumeSpecName: "config-volume") pod "41e7e826-76ea-4c02-87c8-739b0653b66a" (UID: "41e7e826-76ea-4c02-87c8-739b0653b66a"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 17:30:43.962298 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.962275 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41e7e826-76ea-4c02-87c8-739b0653b66a-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "41e7e826-76ea-4c02-87c8-739b0653b66a" (UID: "41e7e826-76ea-4c02-87c8-739b0653b66a"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:30:43.962731 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.962710 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41e7e826-76ea-4c02-87c8-739b0653b66a-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "41e7e826-76ea-4c02-87c8-739b0653b66a" (UID: "41e7e826-76ea-4c02-87c8-739b0653b66a"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 17:30:43.966497 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.966472 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41e7e826-76ea-4c02-87c8-739b0653b66a-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "41e7e826-76ea-4c02-87c8-739b0653b66a" (UID: "41e7e826-76ea-4c02-87c8-739b0653b66a"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 17:30:43.972037 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:43.972015 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41e7e826-76ea-4c02-87c8-739b0653b66a-web-config" (OuterVolumeSpecName: "web-config") pod "41e7e826-76ea-4c02-87c8-739b0653b66a" (UID: "41e7e826-76ea-4c02-87c8-739b0653b66a"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 17:30:44.059629 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.059597 2570 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/41e7e826-76ea-4c02-87c8-739b0653b66a-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-140-185.ec2.internal\" DevicePath \"\"" Jul 27 17:30:44.059629 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.059622 2570 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/41e7e826-76ea-4c02-87c8-739b0653b66a-tls-assets\") on node \"ip-10-0-140-185.ec2.internal\" DevicePath \"\"" Jul 27 17:30:44.059629 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.059633 2570 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/41e7e826-76ea-4c02-87c8-739b0653b66a-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-140-185.ec2.internal\" DevicePath \"\"" Jul 27 17:30:44.059845 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.059642 2570 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/41e7e826-76ea-4c02-87c8-739b0653b66a-web-config\") on node \"ip-10-0-140-185.ec2.internal\" DevicePath \"\"" Jul 27 17:30:44.059845 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.059651 2570 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41e7e826-76ea-4c02-87c8-739b0653b66a-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-140-185.ec2.internal\" DevicePath \"\"" Jul 27 17:30:44.059845 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.059660 2570 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/41e7e826-76ea-4c02-87c8-739b0653b66a-config-out\") on node \"ip-10-0-140-185.ec2.internal\" DevicePath \"\"" Jul 27 17:30:44.059845 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.059668 2570 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41e7e826-76ea-4c02-87c8-739b0653b66a-metrics-client-ca\") on node \"ip-10-0-140-185.ec2.internal\" DevicePath \"\"" Jul 27 17:30:44.059845 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.059676 2570 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/41e7e826-76ea-4c02-87c8-739b0653b66a-cluster-tls-config\") on node \"ip-10-0-140-185.ec2.internal\" DevicePath \"\"" Jul 27 17:30:44.059845 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.059684 2570 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/41e7e826-76ea-4c02-87c8-739b0653b66a-secret-alertmanager-main-tls\") on node \"ip-10-0-140-185.ec2.internal\" DevicePath \"\"" Jul 27 17:30:44.059845 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.059693 2570 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/41e7e826-76ea-4c02-87c8-739b0653b66a-config-volume\") on node \"ip-10-0-140-185.ec2.internal\" DevicePath \"\"" Jul 27 17:30:44.059845 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.059701 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qvbj\" (UniqueName: \"kubernetes.io/projected/41e7e826-76ea-4c02-87c8-739b0653b66a-kube-api-access-5qvbj\") on node \"ip-10-0-140-185.ec2.internal\" DevicePath \"\"" Jul 27 17:30:44.059845 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.059709 2570 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/41e7e826-76ea-4c02-87c8-739b0653b66a-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-140-185.ec2.internal\" DevicePath \"\"" Jul 27 17:30:44.195686 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.195659 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 27 17:30:44.199201 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.199177 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 27 17:30:44.224291 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.224258 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 27 17:30:44.224594 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.224578 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41e7e826-76ea-4c02-87c8-739b0653b66a" containerName="kube-rbac-proxy-web" Jul 27 17:30:44.224683 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.224597 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="41e7e826-76ea-4c02-87c8-739b0653b66a" containerName="kube-rbac-proxy-web" Jul 27 17:30:44.224683 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.224612 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41e7e826-76ea-4c02-87c8-739b0653b66a" containerName="config-reloader" Jul 27 17:30:44.224683 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.224621 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="41e7e826-76ea-4c02-87c8-739b0653b66a" containerName="config-reloader" Jul 27 17:30:44.224683 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.224638 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41e7e826-76ea-4c02-87c8-739b0653b66a" containerName="init-config-reloader" Jul 27 17:30:44.224683 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.224647 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="41e7e826-76ea-4c02-87c8-739b0653b66a" containerName="init-config-reloader" Jul 27 17:30:44.224683 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.224662 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1389cc5-47f4-4804-8c4e-836a5181bd76" containerName="console" Jul 27 17:30:44.224683 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.224670 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1389cc5-47f4-4804-8c4e-836a5181bd76" containerName="console" Jul 27 17:30:44.224683 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.224681 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41e7e826-76ea-4c02-87c8-739b0653b66a" containerName="kube-rbac-proxy-metric" Jul 27 17:30:44.225050 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.224689 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="41e7e826-76ea-4c02-87c8-739b0653b66a" containerName="kube-rbac-proxy-metric" Jul 27 17:30:44.225050 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.224699 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41e7e826-76ea-4c02-87c8-739b0653b66a" containerName="prom-label-proxy" Jul 27 17:30:44.225050 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.224708 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="41e7e826-76ea-4c02-87c8-739b0653b66a" containerName="prom-label-proxy" Jul 27 17:30:44.225050 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.224720 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41e7e826-76ea-4c02-87c8-739b0653b66a" containerName="kube-rbac-proxy" Jul 27 17:30:44.225050 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.224728 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="41e7e826-76ea-4c02-87c8-739b0653b66a" containerName="kube-rbac-proxy" Jul 27 17:30:44.225050 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.224745 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41e7e826-76ea-4c02-87c8-739b0653b66a" containerName="alertmanager" Jul 27 17:30:44.225050 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.224753 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="41e7e826-76ea-4c02-87c8-739b0653b66a" containerName="alertmanager" Jul 27 17:30:44.225050 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.224821 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="41e7e826-76ea-4c02-87c8-739b0653b66a" containerName="prom-label-proxy" Jul 27 17:30:44.225050 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.224835 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="41e7e826-76ea-4c02-87c8-739b0653b66a" containerName="alertmanager" Jul 27 17:30:44.225050 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.224844 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="41e7e826-76ea-4c02-87c8-739b0653b66a" containerName="kube-rbac-proxy-web" Jul 27 17:30:44.225050 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.224854 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1389cc5-47f4-4804-8c4e-836a5181bd76" containerName="console" Jul 27 17:30:44.225050 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.224862 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="41e7e826-76ea-4c02-87c8-739b0653b66a" containerName="config-reloader" Jul 27 17:30:44.225050 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.224869 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="41e7e826-76ea-4c02-87c8-739b0653b66a" containerName="kube-rbac-proxy-metric" Jul 27 17:30:44.225050 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.224879 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="41e7e826-76ea-4c02-87c8-739b0653b66a" containerName="kube-rbac-proxy" Jul 27 17:30:44.228780 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.228759 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:30:44.231252 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.231144 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 27 17:30:44.231252 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.231174 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 27 17:30:44.231252 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.231145 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 27 17:30:44.231252 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.231177 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 27 17:30:44.231252 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.231202 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-fnv5t\"" Jul 27 17:30:44.231252 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.231183 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 27 17:30:44.231252 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.231203 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 27 17:30:44.231685 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.231209 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 27 17:30:44.231685 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.231151 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 27 17:30:44.236792 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.236773 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 27 17:30:44.245733 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.242984 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 27 17:30:44.362347 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.362312 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/afd26584-d3b3-434a-9f1a-a4db8f53259f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"afd26584-d3b3-434a-9f1a-a4db8f53259f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:30:44.362347 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.362344 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/afd26584-d3b3-434a-9f1a-a4db8f53259f-config-out\") pod \"alertmanager-main-0\" (UID: \"afd26584-d3b3-434a-9f1a-a4db8f53259f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:30:44.362562 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.362365 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/afd26584-d3b3-434a-9f1a-a4db8f53259f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"afd26584-d3b3-434a-9f1a-a4db8f53259f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:30:44.362562 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.362387 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/afd26584-d3b3-434a-9f1a-a4db8f53259f-web-config\") pod \"alertmanager-main-0\" (UID: \"afd26584-d3b3-434a-9f1a-a4db8f53259f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:30:44.362562 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.362436 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/afd26584-d3b3-434a-9f1a-a4db8f53259f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"afd26584-d3b3-434a-9f1a-a4db8f53259f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:30:44.362562 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.362477 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/afd26584-d3b3-434a-9f1a-a4db8f53259f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"afd26584-d3b3-434a-9f1a-a4db8f53259f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:30:44.362562 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.362501 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/afd26584-d3b3-434a-9f1a-a4db8f53259f-config-volume\") pod \"alertmanager-main-0\" (UID: \"afd26584-d3b3-434a-9f1a-a4db8f53259f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:30:44.362562 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.362515 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/afd26584-d3b3-434a-9f1a-a4db8f53259f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"afd26584-d3b3-434a-9f1a-a4db8f53259f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:30:44.362562 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.362563 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/afd26584-d3b3-434a-9f1a-a4db8f53259f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"afd26584-d3b3-434a-9f1a-a4db8f53259f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:30:44.362768 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.362588 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/afd26584-d3b3-434a-9f1a-a4db8f53259f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"afd26584-d3b3-434a-9f1a-a4db8f53259f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:30:44.362768 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.362661 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdll4\" (UniqueName: \"kubernetes.io/projected/afd26584-d3b3-434a-9f1a-a4db8f53259f-kube-api-access-zdll4\") pod \"alertmanager-main-0\" (UID: \"afd26584-d3b3-434a-9f1a-a4db8f53259f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:30:44.362768 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.362688 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/afd26584-d3b3-434a-9f1a-a4db8f53259f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"afd26584-d3b3-434a-9f1a-a4db8f53259f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:30:44.362768 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.362713 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/afd26584-d3b3-434a-9f1a-a4db8f53259f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"afd26584-d3b3-434a-9f1a-a4db8f53259f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:30:44.463961 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.463883 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdll4\" (UniqueName: \"kubernetes.io/projected/afd26584-d3b3-434a-9f1a-a4db8f53259f-kube-api-access-zdll4\") pod \"alertmanager-main-0\" (UID: \"afd26584-d3b3-434a-9f1a-a4db8f53259f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:30:44.463961 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.463918 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/afd26584-d3b3-434a-9f1a-a4db8f53259f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"afd26584-d3b3-434a-9f1a-a4db8f53259f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:30:44.463961 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.463936 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/afd26584-d3b3-434a-9f1a-a4db8f53259f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"afd26584-d3b3-434a-9f1a-a4db8f53259f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:30:44.463961 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.463955 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/afd26584-d3b3-434a-9f1a-a4db8f53259f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"afd26584-d3b3-434a-9f1a-a4db8f53259f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:30:44.464256 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.464011 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/afd26584-d3b3-434a-9f1a-a4db8f53259f-config-out\") pod \"alertmanager-main-0\" (UID: \"afd26584-d3b3-434a-9f1a-a4db8f53259f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:30:44.464256 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.464050 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/afd26584-d3b3-434a-9f1a-a4db8f53259f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"afd26584-d3b3-434a-9f1a-a4db8f53259f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:30:44.464256 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.464085 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/afd26584-d3b3-434a-9f1a-a4db8f53259f-web-config\") pod \"alertmanager-main-0\" (UID: \"afd26584-d3b3-434a-9f1a-a4db8f53259f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:30:44.464256 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.464110 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/afd26584-d3b3-434a-9f1a-a4db8f53259f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"afd26584-d3b3-434a-9f1a-a4db8f53259f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:30:44.464256 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.464142 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/afd26584-d3b3-434a-9f1a-a4db8f53259f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"afd26584-d3b3-434a-9f1a-a4db8f53259f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:30:44.464256 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.464186 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/afd26584-d3b3-434a-9f1a-a4db8f53259f-config-volume\") pod \"alertmanager-main-0\" (UID: \"afd26584-d3b3-434a-9f1a-a4db8f53259f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:30:44.464256 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.464213 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/afd26584-d3b3-434a-9f1a-a4db8f53259f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"afd26584-d3b3-434a-9f1a-a4db8f53259f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:30:44.464256 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.464239 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/afd26584-d3b3-434a-9f1a-a4db8f53259f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"afd26584-d3b3-434a-9f1a-a4db8f53259f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:30:44.464665 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.464276 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/afd26584-d3b3-434a-9f1a-a4db8f53259f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"afd26584-d3b3-434a-9f1a-a4db8f53259f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:30:44.464665 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.464463 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/afd26584-d3b3-434a-9f1a-a4db8f53259f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"afd26584-d3b3-434a-9f1a-a4db8f53259f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:30:44.464815 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.464783 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/afd26584-d3b3-434a-9f1a-a4db8f53259f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"afd26584-d3b3-434a-9f1a-a4db8f53259f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:30:44.465317 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.465290 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/afd26584-d3b3-434a-9f1a-a4db8f53259f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"afd26584-d3b3-434a-9f1a-a4db8f53259f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:30:44.467310 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.466949 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/afd26584-d3b3-434a-9f1a-a4db8f53259f-config-out\") pod \"alertmanager-main-0\" (UID: \"afd26584-d3b3-434a-9f1a-a4db8f53259f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:30:44.467310 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.467036 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/afd26584-d3b3-434a-9f1a-a4db8f53259f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"afd26584-d3b3-434a-9f1a-a4db8f53259f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:30:44.467310 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.467108 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/afd26584-d3b3-434a-9f1a-a4db8f53259f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"afd26584-d3b3-434a-9f1a-a4db8f53259f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:30:44.467310 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.467271 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/afd26584-d3b3-434a-9f1a-a4db8f53259f-config-volume\") pod \"alertmanager-main-0\" (UID: \"afd26584-d3b3-434a-9f1a-a4db8f53259f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:30:44.467487 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.467466 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/afd26584-d3b3-434a-9f1a-a4db8f53259f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"afd26584-d3b3-434a-9f1a-a4db8f53259f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:30:44.467543 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.467505 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/afd26584-d3b3-434a-9f1a-a4db8f53259f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"afd26584-d3b3-434a-9f1a-a4db8f53259f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:30:44.467653 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.467636 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/afd26584-d3b3-434a-9f1a-a4db8f53259f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"afd26584-d3b3-434a-9f1a-a4db8f53259f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:30:44.467930 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.467914 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/afd26584-d3b3-434a-9f1a-a4db8f53259f-web-config\") pod \"alertmanager-main-0\" (UID: \"afd26584-d3b3-434a-9f1a-a4db8f53259f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:30:44.468673 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.468651 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/afd26584-d3b3-434a-9f1a-a4db8f53259f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"afd26584-d3b3-434a-9f1a-a4db8f53259f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:30:44.470839 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.470822 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdll4\" (UniqueName: \"kubernetes.io/projected/afd26584-d3b3-434a-9f1a-a4db8f53259f-kube-api-access-zdll4\") pod \"alertmanager-main-0\" (UID: \"afd26584-d3b3-434a-9f1a-a4db8f53259f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:30:44.538456 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.538422 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 27 17:30:44.670823 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.670801 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 27 17:30:44.673060 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:30:44.673032 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafd26584_d3b3_434a_9f1a_a4db8f53259f.slice/crio-becc52c31788809101552983ae3b7829c6ee9e896bef0581197a5788e8713993 WatchSource:0}: Error finding container becc52c31788809101552983ae3b7829c6ee9e896bef0581197a5788e8713993: Status 404 returned error can't find the container with id becc52c31788809101552983ae3b7829c6ee9e896bef0581197a5788e8713993 Jul 27 17:30:44.878487 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.878451 2570 generic.go:356] "Generic (PLEG): container finished" podID="afd26584-d3b3-434a-9f1a-a4db8f53259f" containerID="32b433d9f5412d46e355f89e877f54bd3c937ab36b5333f895c7f42d189580d9" exitCode=0 Jul 27 17:30:44.878669 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.878542 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"afd26584-d3b3-434a-9f1a-a4db8f53259f","Type":"ContainerDied","Data":"32b433d9f5412d46e355f89e877f54bd3c937ab36b5333f895c7f42d189580d9"} Jul 27 17:30:44.878669 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:44.878635 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"afd26584-d3b3-434a-9f1a-a4db8f53259f","Type":"ContainerStarted","Data":"becc52c31788809101552983ae3b7829c6ee9e896bef0581197a5788e8713993"} Jul 27 17:30:45.131149 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:45.131125 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41e7e826-76ea-4c02-87c8-739b0653b66a" path="/var/lib/kubelet/pods/41e7e826-76ea-4c02-87c8-739b0653b66a/volumes" Jul 27 17:30:45.886850 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:45.886772 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"afd26584-d3b3-434a-9f1a-a4db8f53259f","Type":"ContainerStarted","Data":"955737a162247ed3a265967de1cc7438aa053ce4d1b8961994f069251e546b83"} Jul 27 17:30:45.886850 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:45.886834 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"afd26584-d3b3-434a-9f1a-a4db8f53259f","Type":"ContainerStarted","Data":"d9fca4d4cdf158d21084eea03141ac51e54e704d42133deaf2ba20dd191cf271"} Jul 27 17:30:45.887328 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:45.886861 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"afd26584-d3b3-434a-9f1a-a4db8f53259f","Type":"ContainerStarted","Data":"dea7055d10eb5779ee492a0bcec35117b2fc5efa4bb408404add9dc10fad957e"} Jul 27 17:30:45.887328 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:45.886890 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"afd26584-d3b3-434a-9f1a-a4db8f53259f","Type":"ContainerStarted","Data":"0680c5aea913cb5b0d8913352bbd6e973e0935ee1f821e13036a710f4d5546af"} Jul 27 17:30:45.887328 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:45.886915 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"afd26584-d3b3-434a-9f1a-a4db8f53259f","Type":"ContainerStarted","Data":"566a2594b43931f0ff9fa6053aed65b58eedb31ddc685fe0c43482087c95f9b5"} Jul 27 17:30:45.887328 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:45.886940 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"afd26584-d3b3-434a-9f1a-a4db8f53259f","Type":"ContainerStarted","Data":"cab757a719dc361dbac771b8c9c720466bfe12007e19640cc23c60a2e78cfc62"} Jul 27 17:30:45.911454 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:45.911409 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=1.896921024 podStartE2EDuration="1.911355278s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.434254ms" isStatefulPod=false podCreationTimestamp="2026-07-27 17:30:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:30:45.910439922 +0000 UTC m=+163.351654894" watchObservedRunningTime="2026-07-27 17:30:45.911355278 +0000 UTC m=+163.352570252" Jul 27 17:30:47.569615 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:47.569538 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-549d7665ff-sth4s"] Jul 27 17:30:47.571954 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:47.571935 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-549d7665ff-sth4s" Jul 27 17:30:47.574018 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:47.573994 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 27 17:30:47.574138 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:47.573994 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 27 17:30:47.574138 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:47.574065 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-flw78\"" Jul 27 17:30:47.574138 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:47.574094 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 27 17:30:47.574323 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:47.574310 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 27 17:30:47.574477 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:47.574461 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 27 17:30:47.580165 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:47.580142 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 27 17:30:47.585698 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:47.585673 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-549d7665ff-sth4s"] Jul 27 17:30:47.692285 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:47.692253 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4378d963-6214-464e-80f6-89f704ebe2f4-serving-certs-ca-bundle\") pod \"telemeter-client-549d7665ff-sth4s\" (UID: \"4378d963-6214-464e-80f6-89f704ebe2f4\") " pod="openshift-monitoring/telemeter-client-549d7665ff-sth4s" Jul 27 17:30:47.692416 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:47.692301 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqd98\" (UniqueName: \"kubernetes.io/projected/4378d963-6214-464e-80f6-89f704ebe2f4-kube-api-access-xqd98\") pod \"telemeter-client-549d7665ff-sth4s\" (UID: \"4378d963-6214-464e-80f6-89f704ebe2f4\") " pod="openshift-monitoring/telemeter-client-549d7665ff-sth4s" Jul 27 17:30:47.692416 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:47.692345 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4378d963-6214-464e-80f6-89f704ebe2f4-metrics-client-ca\") pod \"telemeter-client-549d7665ff-sth4s\" (UID: \"4378d963-6214-464e-80f6-89f704ebe2f4\") " pod="openshift-monitoring/telemeter-client-549d7665ff-sth4s" Jul 27 17:30:47.692416 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:47.692370 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4378d963-6214-464e-80f6-89f704ebe2f4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-549d7665ff-sth4s\" (UID: \"4378d963-6214-464e-80f6-89f704ebe2f4\") " pod="openshift-monitoring/telemeter-client-549d7665ff-sth4s" Jul 27 17:30:47.692416 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:47.692409 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4378d963-6214-464e-80f6-89f704ebe2f4-federate-client-tls\") pod \"telemeter-client-549d7665ff-sth4s\" (UID: \"4378d963-6214-464e-80f6-89f704ebe2f4\") " pod="openshift-monitoring/telemeter-client-549d7665ff-sth4s" Jul 27 17:30:47.692577 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:47.692433 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4378d963-6214-464e-80f6-89f704ebe2f4-secret-telemeter-client\") pod \"telemeter-client-549d7665ff-sth4s\" (UID: \"4378d963-6214-464e-80f6-89f704ebe2f4\") " pod="openshift-monitoring/telemeter-client-549d7665ff-sth4s" Jul 27 17:30:47.692577 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:47.692450 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4378d963-6214-464e-80f6-89f704ebe2f4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-549d7665ff-sth4s\" (UID: \"4378d963-6214-464e-80f6-89f704ebe2f4\") " pod="openshift-monitoring/telemeter-client-549d7665ff-sth4s" Jul 27 17:30:47.692577 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:47.692498 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4378d963-6214-464e-80f6-89f704ebe2f4-telemeter-client-tls\") pod \"telemeter-client-549d7665ff-sth4s\" (UID: \"4378d963-6214-464e-80f6-89f704ebe2f4\") " pod="openshift-monitoring/telemeter-client-549d7665ff-sth4s" Jul 27 17:30:47.793656 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:47.793621 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4378d963-6214-464e-80f6-89f704ebe2f4-federate-client-tls\") pod \"telemeter-client-549d7665ff-sth4s\" (UID: \"4378d963-6214-464e-80f6-89f704ebe2f4\") " pod="openshift-monitoring/telemeter-client-549d7665ff-sth4s" Jul 27 17:30:47.793816 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:47.793663 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4378d963-6214-464e-80f6-89f704ebe2f4-secret-telemeter-client\") pod \"telemeter-client-549d7665ff-sth4s\" (UID: \"4378d963-6214-464e-80f6-89f704ebe2f4\") " pod="openshift-monitoring/telemeter-client-549d7665ff-sth4s" Jul 27 17:30:47.793816 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:47.793683 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4378d963-6214-464e-80f6-89f704ebe2f4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-549d7665ff-sth4s\" (UID: \"4378d963-6214-464e-80f6-89f704ebe2f4\") " pod="openshift-monitoring/telemeter-client-549d7665ff-sth4s" Jul 27 17:30:47.793816 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:47.793729 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4378d963-6214-464e-80f6-89f704ebe2f4-telemeter-client-tls\") pod \"telemeter-client-549d7665ff-sth4s\" (UID: \"4378d963-6214-464e-80f6-89f704ebe2f4\") " pod="openshift-monitoring/telemeter-client-549d7665ff-sth4s" Jul 27 17:30:47.793816 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:47.793788 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4378d963-6214-464e-80f6-89f704ebe2f4-serving-certs-ca-bundle\") pod \"telemeter-client-549d7665ff-sth4s\" (UID: \"4378d963-6214-464e-80f6-89f704ebe2f4\") " pod="openshift-monitoring/telemeter-client-549d7665ff-sth4s" Jul 27 17:30:47.793980 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:47.793835 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqd98\" (UniqueName: \"kubernetes.io/projected/4378d963-6214-464e-80f6-89f704ebe2f4-kube-api-access-xqd98\") pod \"telemeter-client-549d7665ff-sth4s\" (UID: \"4378d963-6214-464e-80f6-89f704ebe2f4\") " pod="openshift-monitoring/telemeter-client-549d7665ff-sth4s" Jul 27 17:30:47.793980 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:47.793860 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4378d963-6214-464e-80f6-89f704ebe2f4-metrics-client-ca\") pod \"telemeter-client-549d7665ff-sth4s\" (UID: \"4378d963-6214-464e-80f6-89f704ebe2f4\") " pod="openshift-monitoring/telemeter-client-549d7665ff-sth4s" Jul 27 17:30:47.793980 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:47.793888 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4378d963-6214-464e-80f6-89f704ebe2f4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-549d7665ff-sth4s\" (UID: \"4378d963-6214-464e-80f6-89f704ebe2f4\") " pod="openshift-monitoring/telemeter-client-549d7665ff-sth4s" Jul 27 17:30:47.794599 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:47.794567 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4378d963-6214-464e-80f6-89f704ebe2f4-serving-certs-ca-bundle\") pod \"telemeter-client-549d7665ff-sth4s\" (UID: \"4378d963-6214-464e-80f6-89f704ebe2f4\") " pod="openshift-monitoring/telemeter-client-549d7665ff-sth4s" Jul 27 17:30:47.794735 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:47.794727 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4378d963-6214-464e-80f6-89f704ebe2f4-metrics-client-ca\") pod \"telemeter-client-549d7665ff-sth4s\" (UID: \"4378d963-6214-464e-80f6-89f704ebe2f4\") " pod="openshift-monitoring/telemeter-client-549d7665ff-sth4s" Jul 27 17:30:47.794865 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:47.794841 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4378d963-6214-464e-80f6-89f704ebe2f4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-549d7665ff-sth4s\" (UID: \"4378d963-6214-464e-80f6-89f704ebe2f4\") " pod="openshift-monitoring/telemeter-client-549d7665ff-sth4s" Jul 27 17:30:47.796283 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:47.796258 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4378d963-6214-464e-80f6-89f704ebe2f4-telemeter-client-tls\") pod \"telemeter-client-549d7665ff-sth4s\" (UID: \"4378d963-6214-464e-80f6-89f704ebe2f4\") " pod="openshift-monitoring/telemeter-client-549d7665ff-sth4s" Jul 27 17:30:47.796392 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:47.796373 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4378d963-6214-464e-80f6-89f704ebe2f4-secret-telemeter-client\") pod \"telemeter-client-549d7665ff-sth4s\" (UID: \"4378d963-6214-464e-80f6-89f704ebe2f4\") " pod="openshift-monitoring/telemeter-client-549d7665ff-sth4s" Jul 27 17:30:47.796548 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:47.796507 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4378d963-6214-464e-80f6-89f704ebe2f4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-549d7665ff-sth4s\" (UID: \"4378d963-6214-464e-80f6-89f704ebe2f4\") " pod="openshift-monitoring/telemeter-client-549d7665ff-sth4s" Jul 27 17:30:47.796613 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:47.796576 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4378d963-6214-464e-80f6-89f704ebe2f4-federate-client-tls\") pod \"telemeter-client-549d7665ff-sth4s\" (UID: \"4378d963-6214-464e-80f6-89f704ebe2f4\") " pod="openshift-monitoring/telemeter-client-549d7665ff-sth4s" Jul 27 17:30:47.800650 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:47.800629 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqd98\" (UniqueName: \"kubernetes.io/projected/4378d963-6214-464e-80f6-89f704ebe2f4-kube-api-access-xqd98\") pod \"telemeter-client-549d7665ff-sth4s\" (UID: \"4378d963-6214-464e-80f6-89f704ebe2f4\") " pod="openshift-monitoring/telemeter-client-549d7665ff-sth4s" Jul 27 17:30:47.881851 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:47.881772 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-549d7665ff-sth4s" Jul 27 17:30:48.012369 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:48.012345 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-549d7665ff-sth4s"] Jul 27 17:30:48.014127 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:30:48.014098 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4378d963_6214_464e_80f6_89f704ebe2f4.slice/crio-6a4668d46bf62785ea7af2cecc26909cbb7d671652b8f73e3d4c00d8ac0cf3c7 WatchSource:0}: Error finding container 6a4668d46bf62785ea7af2cecc26909cbb7d671652b8f73e3d4c00d8ac0cf3c7: Status 404 returned error can't find the container with id 6a4668d46bf62785ea7af2cecc26909cbb7d671652b8f73e3d4c00d8ac0cf3c7 Jul 27 17:30:48.901352 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:48.901318 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-549d7665ff-sth4s" event={"ID":"4378d963-6214-464e-80f6-89f704ebe2f4","Type":"ContainerStarted","Data":"6a4668d46bf62785ea7af2cecc26909cbb7d671652b8f73e3d4c00d8ac0cf3c7"} Jul 27 17:30:52.923200 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:52.923103 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-549d7665ff-sth4s" event={"ID":"4378d963-6214-464e-80f6-89f704ebe2f4","Type":"ContainerStarted","Data":"12101f9dda8fd169c462ffd1ca6f7d1d586397b6a984160f01e69996e1869c9d"} Jul 27 17:30:52.923200 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:52.923157 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-549d7665ff-sth4s" event={"ID":"4378d963-6214-464e-80f6-89f704ebe2f4","Type":"ContainerStarted","Data":"485dadfc670cf6423dbf2bdb61bf1ab0ea68ce9c4854c53532e516548e299bd0"} Jul 27 17:30:52.923200 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:52.923176 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-549d7665ff-sth4s" event={"ID":"4378d963-6214-464e-80f6-89f704ebe2f4","Type":"ContainerStarted","Data":"266a3f657fe775ce0b1150c8fe2eca101ce778175a222c22545b4cebc92252bb"} Jul 27 17:30:52.946997 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:52.946948 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-549d7665ff-sth4s" podStartSLOduration=1.3707042440000001 podStartE2EDuration="5.946929475s" totalImagesPullingTime="4.576225231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:30:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:30:52.945625596 +0000 UTC m=+170.386840570" watchObservedRunningTime="2026-07-27 17:30:52.946929475 +0000 UTC m=+170.388144449" Jul 27 17:30:53.911869 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:53.911833 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6f76bfddc6-7tgpn"] Jul 27 17:30:53.913946 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:53.913928 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f76bfddc6-7tgpn" Jul 27 17:30:53.926345 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:53.926317 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f76bfddc6-7tgpn"] Jul 27 17:30:53.944001 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:53.943966 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f34392e3-37f6-4c25-85fd-f3a159873d76-console-config\") pod \"console-6f76bfddc6-7tgpn\" (UID: \"f34392e3-37f6-4c25-85fd-f3a159873d76\") " pod="openshift-console/console-6f76bfddc6-7tgpn" Jul 27 17:30:53.944158 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:53.944015 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f34392e3-37f6-4c25-85fd-f3a159873d76-console-serving-cert\") pod \"console-6f76bfddc6-7tgpn\" (UID: \"f34392e3-37f6-4c25-85fd-f3a159873d76\") " pod="openshift-console/console-6f76bfddc6-7tgpn" Jul 27 17:30:53.944158 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:53.944076 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqd8p\" (UniqueName: \"kubernetes.io/projected/f34392e3-37f6-4c25-85fd-f3a159873d76-kube-api-access-bqd8p\") pod \"console-6f76bfddc6-7tgpn\" (UID: \"f34392e3-37f6-4c25-85fd-f3a159873d76\") " pod="openshift-console/console-6f76bfddc6-7tgpn" Jul 27 17:30:53.944282 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:53.944251 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f34392e3-37f6-4c25-85fd-f3a159873d76-service-ca\") pod \"console-6f76bfddc6-7tgpn\" (UID: \"f34392e3-37f6-4c25-85fd-f3a159873d76\") " pod="openshift-console/console-6f76bfddc6-7tgpn" Jul 27 17:30:53.944346 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:53.944333 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f34392e3-37f6-4c25-85fd-f3a159873d76-oauth-serving-cert\") pod \"console-6f76bfddc6-7tgpn\" (UID: \"f34392e3-37f6-4c25-85fd-f3a159873d76\") " pod="openshift-console/console-6f76bfddc6-7tgpn" Jul 27 17:30:53.944402 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:53.944365 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f34392e3-37f6-4c25-85fd-f3a159873d76-console-oauth-config\") pod \"console-6f76bfddc6-7tgpn\" (UID: \"f34392e3-37f6-4c25-85fd-f3a159873d76\") " pod="openshift-console/console-6f76bfddc6-7tgpn" Jul 27 17:30:53.944402 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:53.944387 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f34392e3-37f6-4c25-85fd-f3a159873d76-trusted-ca-bundle\") pod \"console-6f76bfddc6-7tgpn\" (UID: \"f34392e3-37f6-4c25-85fd-f3a159873d76\") " pod="openshift-console/console-6f76bfddc6-7tgpn" Jul 27 17:30:54.045847 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:54.045805 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f34392e3-37f6-4c25-85fd-f3a159873d76-console-config\") pod \"console-6f76bfddc6-7tgpn\" (UID: \"f34392e3-37f6-4c25-85fd-f3a159873d76\") " pod="openshift-console/console-6f76bfddc6-7tgpn" Jul 27 17:30:54.045847 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:54.045852 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f34392e3-37f6-4c25-85fd-f3a159873d76-console-serving-cert\") pod \"console-6f76bfddc6-7tgpn\" (UID: \"f34392e3-37f6-4c25-85fd-f3a159873d76\") " pod="openshift-console/console-6f76bfddc6-7tgpn" Jul 27 17:30:54.046099 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:54.045895 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqd8p\" (UniqueName: \"kubernetes.io/projected/f34392e3-37f6-4c25-85fd-f3a159873d76-kube-api-access-bqd8p\") pod \"console-6f76bfddc6-7tgpn\" (UID: \"f34392e3-37f6-4c25-85fd-f3a159873d76\") " pod="openshift-console/console-6f76bfddc6-7tgpn" Jul 27 17:30:54.046099 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:54.045924 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f34392e3-37f6-4c25-85fd-f3a159873d76-service-ca\") pod \"console-6f76bfddc6-7tgpn\" (UID: \"f34392e3-37f6-4c25-85fd-f3a159873d76\") " pod="openshift-console/console-6f76bfddc6-7tgpn" Jul 27 17:30:54.046099 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:54.045961 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f34392e3-37f6-4c25-85fd-f3a159873d76-oauth-serving-cert\") pod \"console-6f76bfddc6-7tgpn\" (UID: \"f34392e3-37f6-4c25-85fd-f3a159873d76\") " pod="openshift-console/console-6f76bfddc6-7tgpn" Jul 27 17:30:54.046099 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:54.045985 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f34392e3-37f6-4c25-85fd-f3a159873d76-console-oauth-config\") pod \"console-6f76bfddc6-7tgpn\" (UID: \"f34392e3-37f6-4c25-85fd-f3a159873d76\") " pod="openshift-console/console-6f76bfddc6-7tgpn" Jul 27 17:30:54.046099 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:54.046007 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f34392e3-37f6-4c25-85fd-f3a159873d76-trusted-ca-bundle\") pod \"console-6f76bfddc6-7tgpn\" (UID: \"f34392e3-37f6-4c25-85fd-f3a159873d76\") " pod="openshift-console/console-6f76bfddc6-7tgpn" Jul 27 17:30:54.046680 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:54.046658 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f34392e3-37f6-4c25-85fd-f3a159873d76-oauth-serving-cert\") pod \"console-6f76bfddc6-7tgpn\" (UID: \"f34392e3-37f6-4c25-85fd-f3a159873d76\") " pod="openshift-console/console-6f76bfddc6-7tgpn" Jul 27 17:30:54.046794 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:54.046671 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f34392e3-37f6-4c25-85fd-f3a159873d76-service-ca\") pod \"console-6f76bfddc6-7tgpn\" (UID: \"f34392e3-37f6-4c25-85fd-f3a159873d76\") " pod="openshift-console/console-6f76bfddc6-7tgpn" Jul 27 17:30:54.046839 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:54.046793 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f34392e3-37f6-4c25-85fd-f3a159873d76-trusted-ca-bundle\") pod \"console-6f76bfddc6-7tgpn\" (UID: \"f34392e3-37f6-4c25-85fd-f3a159873d76\") " pod="openshift-console/console-6f76bfddc6-7tgpn" Jul 27 17:30:54.046873 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:54.046843 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f34392e3-37f6-4c25-85fd-f3a159873d76-console-config\") pod \"console-6f76bfddc6-7tgpn\" (UID: \"f34392e3-37f6-4c25-85fd-f3a159873d76\") " pod="openshift-console/console-6f76bfddc6-7tgpn" Jul 27 17:30:54.048860 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:54.048840 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f34392e3-37f6-4c25-85fd-f3a159873d76-console-serving-cert\") pod \"console-6f76bfddc6-7tgpn\" (UID: \"f34392e3-37f6-4c25-85fd-f3a159873d76\") " pod="openshift-console/console-6f76bfddc6-7tgpn" Jul 27 17:30:54.048948 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:54.048856 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f34392e3-37f6-4c25-85fd-f3a159873d76-console-oauth-config\") pod \"console-6f76bfddc6-7tgpn\" (UID: \"f34392e3-37f6-4c25-85fd-f3a159873d76\") " pod="openshift-console/console-6f76bfddc6-7tgpn" Jul 27 17:30:54.052971 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:54.052946 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqd8p\" (UniqueName: \"kubernetes.io/projected/f34392e3-37f6-4c25-85fd-f3a159873d76-kube-api-access-bqd8p\") pod \"console-6f76bfddc6-7tgpn\" (UID: \"f34392e3-37f6-4c25-85fd-f3a159873d76\") " pod="openshift-console/console-6f76bfddc6-7tgpn" Jul 27 17:30:54.222854 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:54.222764 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f76bfddc6-7tgpn" Jul 27 17:30:54.352463 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:54.352395 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f76bfddc6-7tgpn"] Jul 27 17:30:54.355274 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:30:54.355245 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf34392e3_37f6_4c25_85fd_f3a159873d76.slice/crio-c54344b109e17aef96a681ff48935dabd8042a40f4199b3727a5b9d4f79b9d58 WatchSource:0}: Error finding container c54344b109e17aef96a681ff48935dabd8042a40f4199b3727a5b9d4f79b9d58: Status 404 returned error can't find the container with id c54344b109e17aef96a681ff48935dabd8042a40f4199b3727a5b9d4f79b9d58 Jul 27 17:30:54.937825 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:54.937790 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f76bfddc6-7tgpn" event={"ID":"f34392e3-37f6-4c25-85fd-f3a159873d76","Type":"ContainerStarted","Data":"adbaabb1697561226aa3c868ce53bcaae209eabab2375b370121dbc02cd1e964"} Jul 27 17:30:54.938205 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:54.937844 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f76bfddc6-7tgpn" event={"ID":"f34392e3-37f6-4c25-85fd-f3a159873d76","Type":"ContainerStarted","Data":"c54344b109e17aef96a681ff48935dabd8042a40f4199b3727a5b9d4f79b9d58"} Jul 27 17:30:54.954275 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:30:54.954224 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6f76bfddc6-7tgpn" podStartSLOduration=1.954204276 podStartE2EDuration="1.954204276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:30:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:30:54.953152588 +0000 UTC m=+172.394367564" watchObservedRunningTime="2026-07-27 17:30:54.954204276 +0000 UTC m=+172.395419248" Jul 27 17:31:04.223880 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:31:04.223843 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6f76bfddc6-7tgpn" Jul 27 17:31:04.224254 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:31:04.223897 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6f76bfddc6-7tgpn" Jul 27 17:31:04.228787 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:31:04.228762 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6f76bfddc6-7tgpn" Jul 27 17:31:04.980473 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:31:04.980446 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6f76bfddc6-7tgpn" Jul 27 17:31:05.035474 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:31:05.035440 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7848d54945-sg2cv"] Jul 27 17:31:30.054755 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:31:30.054710 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7848d54945-sg2cv" podUID="c3629ded-e6b6-4068-9490-484e343b7a38" containerName="console" containerID="cri-o://d5115a45e56a7747a324feb4d933f0b7e8b6e6f36af7b6f1d9e7b08b974ae761" gracePeriod=15 Jul 27 17:31:30.300976 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:31:30.300953 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7848d54945-sg2cv" Jul 27 17:31:30.462659 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:31:30.462562 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c3629ded-e6b6-4068-9490-484e343b7a38-oauth-serving-cert\") pod \"c3629ded-e6b6-4068-9490-484e343b7a38\" (UID: \"c3629ded-e6b6-4068-9490-484e343b7a38\") " Jul 27 17:31:30.462659 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:31:30.462611 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3629ded-e6b6-4068-9490-484e343b7a38-trusted-ca-bundle\") pod \"c3629ded-e6b6-4068-9490-484e343b7a38\" (UID: \"c3629ded-e6b6-4068-9490-484e343b7a38\") " Jul 27 17:31:30.462879 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:31:30.462668 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dd9wl\" (UniqueName: \"kubernetes.io/projected/c3629ded-e6b6-4068-9490-484e343b7a38-kube-api-access-dd9wl\") pod \"c3629ded-e6b6-4068-9490-484e343b7a38\" (UID: \"c3629ded-e6b6-4068-9490-484e343b7a38\") " Jul 27 17:31:30.462879 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:31:30.462697 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c3629ded-e6b6-4068-9490-484e343b7a38-console-serving-cert\") pod \"c3629ded-e6b6-4068-9490-484e343b7a38\" (UID: \"c3629ded-e6b6-4068-9490-484e343b7a38\") " Jul 27 17:31:30.462879 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:31:30.462864 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c3629ded-e6b6-4068-9490-484e343b7a38-console-oauth-config\") pod \"c3629ded-e6b6-4068-9490-484e343b7a38\" (UID: \"c3629ded-e6b6-4068-9490-484e343b7a38\") " Jul 27 17:31:30.463033 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:31:30.462922 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c3629ded-e6b6-4068-9490-484e343b7a38-service-ca\") pod \"c3629ded-e6b6-4068-9490-484e343b7a38\" (UID: \"c3629ded-e6b6-4068-9490-484e343b7a38\") " Jul 27 17:31:30.463033 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:31:30.462951 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c3629ded-e6b6-4068-9490-484e343b7a38-console-config\") pod \"c3629ded-e6b6-4068-9490-484e343b7a38\" (UID: \"c3629ded-e6b6-4068-9490-484e343b7a38\") " Jul 27 17:31:30.463033 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:31:30.462951 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3629ded-e6b6-4068-9490-484e343b7a38-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c3629ded-e6b6-4068-9490-484e343b7a38" (UID: "c3629ded-e6b6-4068-9490-484e343b7a38"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:31:30.463195 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:31:30.463166 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3629ded-e6b6-4068-9490-484e343b7a38-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "c3629ded-e6b6-4068-9490-484e343b7a38" (UID: "c3629ded-e6b6-4068-9490-484e343b7a38"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:31:30.463246 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:31:30.463199 2570 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c3629ded-e6b6-4068-9490-484e343b7a38-oauth-serving-cert\") on node \"ip-10-0-140-185.ec2.internal\" DevicePath \"\"" Jul 27 17:31:30.463319 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:31:30.463296 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3629ded-e6b6-4068-9490-484e343b7a38-service-ca" (OuterVolumeSpecName: "service-ca") pod "c3629ded-e6b6-4068-9490-484e343b7a38" (UID: "c3629ded-e6b6-4068-9490-484e343b7a38"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:31:30.463376 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:31:30.463344 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3629ded-e6b6-4068-9490-484e343b7a38-console-config" (OuterVolumeSpecName: "console-config") pod "c3629ded-e6b6-4068-9490-484e343b7a38" (UID: "c3629ded-e6b6-4068-9490-484e343b7a38"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:31:30.464895 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:31:30.464866 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3629ded-e6b6-4068-9490-484e343b7a38-kube-api-access-dd9wl" (OuterVolumeSpecName: "kube-api-access-dd9wl") pod "c3629ded-e6b6-4068-9490-484e343b7a38" (UID: "c3629ded-e6b6-4068-9490-484e343b7a38"). InnerVolumeSpecName "kube-api-access-dd9wl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:31:30.464895 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:31:30.464876 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3629ded-e6b6-4068-9490-484e343b7a38-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c3629ded-e6b6-4068-9490-484e343b7a38" (UID: "c3629ded-e6b6-4068-9490-484e343b7a38"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 17:31:30.465041 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:31:30.464919 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3629ded-e6b6-4068-9490-484e343b7a38-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c3629ded-e6b6-4068-9490-484e343b7a38" (UID: "c3629ded-e6b6-4068-9490-484e343b7a38"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 17:31:30.564373 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:31:30.564331 2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3629ded-e6b6-4068-9490-484e343b7a38-trusted-ca-bundle\") on node \"ip-10-0-140-185.ec2.internal\" DevicePath \"\"" Jul 27 17:31:30.564373 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:31:30.564365 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dd9wl\" (UniqueName: \"kubernetes.io/projected/c3629ded-e6b6-4068-9490-484e343b7a38-kube-api-access-dd9wl\") on node \"ip-10-0-140-185.ec2.internal\" DevicePath \"\"" Jul 27 17:31:30.564373 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:31:30.564378 2570 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c3629ded-e6b6-4068-9490-484e343b7a38-console-serving-cert\") on node \"ip-10-0-140-185.ec2.internal\" DevicePath \"\"" Jul 27 17:31:30.564651 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:31:30.564389 2570 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c3629ded-e6b6-4068-9490-484e343b7a38-console-oauth-config\") on node \"ip-10-0-140-185.ec2.internal\" DevicePath \"\"" Jul 27 17:31:30.564651 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:31:30.564402 2570 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c3629ded-e6b6-4068-9490-484e343b7a38-service-ca\") on node \"ip-10-0-140-185.ec2.internal\" DevicePath \"\"" Jul 27 17:31:30.564651 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:31:30.564414 2570 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c3629ded-e6b6-4068-9490-484e343b7a38-console-config\") on node \"ip-10-0-140-185.ec2.internal\" DevicePath \"\"" Jul 27 17:31:31.080696 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:31:31.080661 2570 generic.go:356] "Generic (PLEG): container finished" podID="c3629ded-e6b6-4068-9490-484e343b7a38" containerID="d5115a45e56a7747a324feb4d933f0b7e8b6e6f36af7b6f1d9e7b08b974ae761" exitCode=0 Jul 27 17:31:31.081125 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:31:31.080721 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7848d54945-sg2cv" Jul 27 17:31:31.081125 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:31:31.080734 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7848d54945-sg2cv" event={"ID":"c3629ded-e6b6-4068-9490-484e343b7a38","Type":"ContainerDied","Data":"d5115a45e56a7747a324feb4d933f0b7e8b6e6f36af7b6f1d9e7b08b974ae761"} Jul 27 17:31:31.081125 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:31:31.080822 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7848d54945-sg2cv" event={"ID":"c3629ded-e6b6-4068-9490-484e343b7a38","Type":"ContainerDied","Data":"cef1cdb150b7d3d6944d664ca4668e6448406270d88f6f522fe77506ce7d6059"} Jul 27 17:31:31.081125 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:31:31.080850 2570 scope.go:117] "RemoveContainer" containerID="d5115a45e56a7747a324feb4d933f0b7e8b6e6f36af7b6f1d9e7b08b974ae761" Jul 27 17:31:31.088908 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:31:31.088891 2570 scope.go:117] "RemoveContainer" containerID="d5115a45e56a7747a324feb4d933f0b7e8b6e6f36af7b6f1d9e7b08b974ae761" Jul 27 17:31:31.089163 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:31:31.089145 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5115a45e56a7747a324feb4d933f0b7e8b6e6f36af7b6f1d9e7b08b974ae761\": container with ID starting with d5115a45e56a7747a324feb4d933f0b7e8b6e6f36af7b6f1d9e7b08b974ae761 not found: ID does not exist" containerID="d5115a45e56a7747a324feb4d933f0b7e8b6e6f36af7b6f1d9e7b08b974ae761" Jul 27 17:31:31.089221 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:31:31.089174 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5115a45e56a7747a324feb4d933f0b7e8b6e6f36af7b6f1d9e7b08b974ae761"} err="failed to get container status \"d5115a45e56a7747a324feb4d933f0b7e8b6e6f36af7b6f1d9e7b08b974ae761\": rpc error: code = NotFound desc = could not find container \"d5115a45e56a7747a324feb4d933f0b7e8b6e6f36af7b6f1d9e7b08b974ae761\": container with ID starting with d5115a45e56a7747a324feb4d933f0b7e8b6e6f36af7b6f1d9e7b08b974ae761 not found: ID does not exist" Jul 27 17:31:31.100256 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:31:31.100228 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7848d54945-sg2cv"] Jul 27 17:31:31.104368 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:31:31.104346 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7848d54945-sg2cv"] Jul 27 17:31:31.128111 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:31:31.128073 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3629ded-e6b6-4068-9490-484e343b7a38" path="/var/lib/kubelet/pods/c3629ded-e6b6-4068-9490-484e343b7a38/volumes" Jul 27 17:32:01.278636 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:01.278607 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-66b76948bc-8ww6t"] Jul 27 17:32:01.279108 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:01.278907 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3629ded-e6b6-4068-9490-484e343b7a38" containerName="console" Jul 27 17:32:01.279108 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:01.278919 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3629ded-e6b6-4068-9490-484e343b7a38" containerName="console" Jul 27 17:32:01.279108 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:01.278974 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3629ded-e6b6-4068-9490-484e343b7a38" containerName="console" Jul 27 17:32:01.281832 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:01.281816 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66b76948bc-8ww6t" Jul 27 17:32:01.295367 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:01.295286 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66b76948bc-8ww6t"] Jul 27 17:32:01.404949 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:01.404906 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fb6cb5d1-d726-46e3-87af-d83e6af93452-console-config\") pod \"console-66b76948bc-8ww6t\" (UID: \"fb6cb5d1-d726-46e3-87af-d83e6af93452\") " pod="openshift-console/console-66b76948bc-8ww6t" Jul 27 17:32:01.405123 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:01.404961 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fb6cb5d1-d726-46e3-87af-d83e6af93452-service-ca\") pod \"console-66b76948bc-8ww6t\" (UID: \"fb6cb5d1-d726-46e3-87af-d83e6af93452\") " pod="openshift-console/console-66b76948bc-8ww6t" Jul 27 17:32:01.405123 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:01.404981 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fb6cb5d1-d726-46e3-87af-d83e6af93452-console-oauth-config\") pod \"console-66b76948bc-8ww6t\" (UID: \"fb6cb5d1-d726-46e3-87af-d83e6af93452\") " pod="openshift-console/console-66b76948bc-8ww6t" Jul 27 17:32:01.405123 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:01.405057 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmch5\" (UniqueName: \"kubernetes.io/projected/fb6cb5d1-d726-46e3-87af-d83e6af93452-kube-api-access-vmch5\") pod \"console-66b76948bc-8ww6t\" (UID: \"fb6cb5d1-d726-46e3-87af-d83e6af93452\") " pod="openshift-console/console-66b76948bc-8ww6t" Jul 27 17:32:01.405123 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:01.405099 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb6cb5d1-d726-46e3-87af-d83e6af93452-trusted-ca-bundle\") pod \"console-66b76948bc-8ww6t\" (UID: \"fb6cb5d1-d726-46e3-87af-d83e6af93452\") " pod="openshift-console/console-66b76948bc-8ww6t" Jul 27 17:32:01.405252 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:01.405128 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fb6cb5d1-d726-46e3-87af-d83e6af93452-oauth-serving-cert\") pod \"console-66b76948bc-8ww6t\" (UID: \"fb6cb5d1-d726-46e3-87af-d83e6af93452\") " pod="openshift-console/console-66b76948bc-8ww6t" Jul 27 17:32:01.405252 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:01.405167 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb6cb5d1-d726-46e3-87af-d83e6af93452-console-serving-cert\") pod \"console-66b76948bc-8ww6t\" (UID: \"fb6cb5d1-d726-46e3-87af-d83e6af93452\") " pod="openshift-console/console-66b76948bc-8ww6t" Jul 27 17:32:01.505867 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:01.505833 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fb6cb5d1-d726-46e3-87af-d83e6af93452-console-config\") pod \"console-66b76948bc-8ww6t\" (UID: \"fb6cb5d1-d726-46e3-87af-d83e6af93452\") " pod="openshift-console/console-66b76948bc-8ww6t" Jul 27 17:32:01.506040 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:01.505882 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fb6cb5d1-d726-46e3-87af-d83e6af93452-service-ca\") pod \"console-66b76948bc-8ww6t\" (UID: \"fb6cb5d1-d726-46e3-87af-d83e6af93452\") " pod="openshift-console/console-66b76948bc-8ww6t" Jul 27 17:32:01.506040 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:01.505907 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fb6cb5d1-d726-46e3-87af-d83e6af93452-console-oauth-config\") pod \"console-66b76948bc-8ww6t\" (UID: \"fb6cb5d1-d726-46e3-87af-d83e6af93452\") " pod="openshift-console/console-66b76948bc-8ww6t" Jul 27 17:32:01.506040 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:01.505951 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmch5\" (UniqueName: \"kubernetes.io/projected/fb6cb5d1-d726-46e3-87af-d83e6af93452-kube-api-access-vmch5\") pod \"console-66b76948bc-8ww6t\" (UID: \"fb6cb5d1-d726-46e3-87af-d83e6af93452\") " pod="openshift-console/console-66b76948bc-8ww6t" Jul 27 17:32:01.506040 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:01.505979 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb6cb5d1-d726-46e3-87af-d83e6af93452-trusted-ca-bundle\") pod \"console-66b76948bc-8ww6t\" (UID: \"fb6cb5d1-d726-46e3-87af-d83e6af93452\") " pod="openshift-console/console-66b76948bc-8ww6t" Jul 27 17:32:01.506214 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:01.506056 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fb6cb5d1-d726-46e3-87af-d83e6af93452-oauth-serving-cert\") pod \"console-66b76948bc-8ww6t\" (UID: \"fb6cb5d1-d726-46e3-87af-d83e6af93452\") " pod="openshift-console/console-66b76948bc-8ww6t" Jul 27 17:32:01.506214 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:01.506099 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb6cb5d1-d726-46e3-87af-d83e6af93452-console-serving-cert\") pod \"console-66b76948bc-8ww6t\" (UID: \"fb6cb5d1-d726-46e3-87af-d83e6af93452\") " pod="openshift-console/console-66b76948bc-8ww6t" Jul 27 17:32:01.506691 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:01.506667 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fb6cb5d1-d726-46e3-87af-d83e6af93452-service-ca\") pod \"console-66b76948bc-8ww6t\" (UID: \"fb6cb5d1-d726-46e3-87af-d83e6af93452\") " pod="openshift-console/console-66b76948bc-8ww6t" Jul 27 17:32:01.506792 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:01.506741 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fb6cb5d1-d726-46e3-87af-d83e6af93452-console-config\") pod \"console-66b76948bc-8ww6t\" (UID: \"fb6cb5d1-d726-46e3-87af-d83e6af93452\") " pod="openshift-console/console-66b76948bc-8ww6t" Jul 27 17:32:01.506837 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:01.506796 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fb6cb5d1-d726-46e3-87af-d83e6af93452-oauth-serving-cert\") pod \"console-66b76948bc-8ww6t\" (UID: \"fb6cb5d1-d726-46e3-87af-d83e6af93452\") " pod="openshift-console/console-66b76948bc-8ww6t" Jul 27 17:32:01.506934 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:01.506909 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb6cb5d1-d726-46e3-87af-d83e6af93452-trusted-ca-bundle\") pod \"console-66b76948bc-8ww6t\" (UID: \"fb6cb5d1-d726-46e3-87af-d83e6af93452\") " pod="openshift-console/console-66b76948bc-8ww6t" Jul 27 17:32:01.508451 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:01.508428 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fb6cb5d1-d726-46e3-87af-d83e6af93452-console-oauth-config\") pod \"console-66b76948bc-8ww6t\" (UID: \"fb6cb5d1-d726-46e3-87af-d83e6af93452\") " pod="openshift-console/console-66b76948bc-8ww6t" Jul 27 17:32:01.508531 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:01.508506 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb6cb5d1-d726-46e3-87af-d83e6af93452-console-serving-cert\") pod \"console-66b76948bc-8ww6t\" (UID: \"fb6cb5d1-d726-46e3-87af-d83e6af93452\") " pod="openshift-console/console-66b76948bc-8ww6t" Jul 27 17:32:01.513616 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:01.513591 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmch5\" (UniqueName: \"kubernetes.io/projected/fb6cb5d1-d726-46e3-87af-d83e6af93452-kube-api-access-vmch5\") pod \"console-66b76948bc-8ww6t\" (UID: \"fb6cb5d1-d726-46e3-87af-d83e6af93452\") " pod="openshift-console/console-66b76948bc-8ww6t" Jul 27 17:32:01.590773 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:01.590672 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66b76948bc-8ww6t" Jul 27 17:32:01.725940 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:01.725878 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66b76948bc-8ww6t"] Jul 27 17:32:01.728601 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:32:01.728572 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb6cb5d1_d726_46e3_87af_d83e6af93452.slice/crio-137dd298a7b484bd0bfdbe35c098935e1cce400450de465ae219bf3fd4b55e74 WatchSource:0}: Error finding container 137dd298a7b484bd0bfdbe35c098935e1cce400450de465ae219bf3fd4b55e74: Status 404 returned error can't find the container with id 137dd298a7b484bd0bfdbe35c098935e1cce400450de465ae219bf3fd4b55e74 Jul 27 17:32:02.207060 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:02.207025 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66b76948bc-8ww6t" event={"ID":"fb6cb5d1-d726-46e3-87af-d83e6af93452","Type":"ContainerStarted","Data":"9fc254ed474d6479d2ef3792cdad1f34fa57febb9812d7c55ad63c9acc970f9b"} Jul 27 17:32:02.207227 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:02.207074 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66b76948bc-8ww6t" event={"ID":"fb6cb5d1-d726-46e3-87af-d83e6af93452","Type":"ContainerStarted","Data":"137dd298a7b484bd0bfdbe35c098935e1cce400450de465ae219bf3fd4b55e74"} Jul 27 17:32:02.228697 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:02.228646 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-66b76948bc-8ww6t" podStartSLOduration=1.228627786 podStartE2EDuration="1.228627786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:32:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:32:02.227233365 +0000 UTC m=+239.668448343" watchObservedRunningTime="2026-07-27 17:32:02.228627786 +0000 UTC m=+239.669842765" Jul 27 17:32:11.591003 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:11.590969 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-66b76948bc-8ww6t" Jul 27 17:32:11.591460 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:11.591031 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-66b76948bc-8ww6t" Jul 27 17:32:11.595775 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:11.595752 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-66b76948bc-8ww6t" Jul 27 17:32:12.249358 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:12.249330 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-66b76948bc-8ww6t" Jul 27 17:32:12.291894 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:12.291861 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f76bfddc6-7tgpn"] Jul 27 17:32:37.312569 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:37.312449 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6f76bfddc6-7tgpn" podUID="f34392e3-37f6-4c25-85fd-f3a159873d76" containerName="console" containerID="cri-o://adbaabb1697561226aa3c868ce53bcaae209eabab2375b370121dbc02cd1e964" gracePeriod=15 Jul 27 17:32:37.550186 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:37.550158 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f76bfddc6-7tgpn" Jul 27 17:32:37.708583 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:37.708483 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f34392e3-37f6-4c25-85fd-f3a159873d76-console-serving-cert\") pod \"f34392e3-37f6-4c25-85fd-f3a159873d76\" (UID: \"f34392e3-37f6-4c25-85fd-f3a159873d76\") " Jul 27 17:32:37.708583 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:37.708536 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f34392e3-37f6-4c25-85fd-f3a159873d76-console-config\") pod \"f34392e3-37f6-4c25-85fd-f3a159873d76\" (UID: \"f34392e3-37f6-4c25-85fd-f3a159873d76\") " Jul 27 17:32:37.708583 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:37.708556 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f34392e3-37f6-4c25-85fd-f3a159873d76-service-ca\") pod \"f34392e3-37f6-4c25-85fd-f3a159873d76\" (UID: \"f34392e3-37f6-4c25-85fd-f3a159873d76\") " Jul 27 17:32:37.708583 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:37.708571 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f34392e3-37f6-4c25-85fd-f3a159873d76-oauth-serving-cert\") pod \"f34392e3-37f6-4c25-85fd-f3a159873d76\" (UID: \"f34392e3-37f6-4c25-85fd-f3a159873d76\") " Jul 27 17:32:37.708902 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:37.708599 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f34392e3-37f6-4c25-85fd-f3a159873d76-console-oauth-config\") pod \"f34392e3-37f6-4c25-85fd-f3a159873d76\" (UID: \"f34392e3-37f6-4c25-85fd-f3a159873d76\") " Jul 27 17:32:37.708902 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:37.708630 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f34392e3-37f6-4c25-85fd-f3a159873d76-trusted-ca-bundle\") pod \"f34392e3-37f6-4c25-85fd-f3a159873d76\" (UID: \"f34392e3-37f6-4c25-85fd-f3a159873d76\") " Jul 27 17:32:37.708902 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:37.708660 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bqd8p\" (UniqueName: \"kubernetes.io/projected/f34392e3-37f6-4c25-85fd-f3a159873d76-kube-api-access-bqd8p\") pod \"f34392e3-37f6-4c25-85fd-f3a159873d76\" (UID: \"f34392e3-37f6-4c25-85fd-f3a159873d76\") " Jul 27 17:32:37.709119 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:37.709013 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f34392e3-37f6-4c25-85fd-f3a159873d76-console-config" (OuterVolumeSpecName: "console-config") pod "f34392e3-37f6-4c25-85fd-f3a159873d76" (UID: "f34392e3-37f6-4c25-85fd-f3a159873d76"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:32:37.709119 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:37.709022 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f34392e3-37f6-4c25-85fd-f3a159873d76-service-ca" (OuterVolumeSpecName: "service-ca") pod "f34392e3-37f6-4c25-85fd-f3a159873d76" (UID: "f34392e3-37f6-4c25-85fd-f3a159873d76"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:32:37.709119 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:37.709059 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f34392e3-37f6-4c25-85fd-f3a159873d76-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f34392e3-37f6-4c25-85fd-f3a159873d76" (UID: "f34392e3-37f6-4c25-85fd-f3a159873d76"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:32:37.709273 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:37.709117 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f34392e3-37f6-4c25-85fd-f3a159873d76-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "f34392e3-37f6-4c25-85fd-f3a159873d76" (UID: "f34392e3-37f6-4c25-85fd-f3a159873d76"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:32:37.715942 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:37.715903 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f34392e3-37f6-4c25-85fd-f3a159873d76-kube-api-access-bqd8p" (OuterVolumeSpecName: "kube-api-access-bqd8p") pod "f34392e3-37f6-4c25-85fd-f3a159873d76" (UID: "f34392e3-37f6-4c25-85fd-f3a159873d76"). InnerVolumeSpecName "kube-api-access-bqd8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:32:37.715942 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:37.715912 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f34392e3-37f6-4c25-85fd-f3a159873d76-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f34392e3-37f6-4c25-85fd-f3a159873d76" (UID: "f34392e3-37f6-4c25-85fd-f3a159873d76"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 17:32:37.716166 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:37.716113 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f34392e3-37f6-4c25-85fd-f3a159873d76-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f34392e3-37f6-4c25-85fd-f3a159873d76" (UID: "f34392e3-37f6-4c25-85fd-f3a159873d76"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 17:32:37.809947 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:37.809908 2570 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f34392e3-37f6-4c25-85fd-f3a159873d76-service-ca\") on node \"ip-10-0-140-185.ec2.internal\" DevicePath \"\"" Jul 27 17:32:37.809947 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:37.809942 2570 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f34392e3-37f6-4c25-85fd-f3a159873d76-oauth-serving-cert\") on node \"ip-10-0-140-185.ec2.internal\" DevicePath \"\"" Jul 27 17:32:37.809947 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:37.809956 2570 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f34392e3-37f6-4c25-85fd-f3a159873d76-console-oauth-config\") on node \"ip-10-0-140-185.ec2.internal\" DevicePath \"\"" Jul 27 17:32:37.810181 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:37.809968 2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f34392e3-37f6-4c25-85fd-f3a159873d76-trusted-ca-bundle\") on node \"ip-10-0-140-185.ec2.internal\" DevicePath \"\"" Jul 27 17:32:37.810181 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:37.809980 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqd8p\" (UniqueName: \"kubernetes.io/projected/f34392e3-37f6-4c25-85fd-f3a159873d76-kube-api-access-bqd8p\") on node \"ip-10-0-140-185.ec2.internal\" DevicePath \"\"" Jul 27 17:32:37.810181 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:37.809992 2570 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f34392e3-37f6-4c25-85fd-f3a159873d76-console-serving-cert\") on node \"ip-10-0-140-185.ec2.internal\" DevicePath \"\"" Jul 27 17:32:37.810181 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:37.810002 2570 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f34392e3-37f6-4c25-85fd-f3a159873d76-console-config\") on node \"ip-10-0-140-185.ec2.internal\" DevicePath \"\"" Jul 27 17:32:38.340640 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:38.340603 2570 generic.go:356] "Generic (PLEG): container finished" podID="f34392e3-37f6-4c25-85fd-f3a159873d76" containerID="adbaabb1697561226aa3c868ce53bcaae209eabab2375b370121dbc02cd1e964" exitCode=0 Jul 27 17:32:38.341071 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:38.340657 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f76bfddc6-7tgpn" event={"ID":"f34392e3-37f6-4c25-85fd-f3a159873d76","Type":"ContainerDied","Data":"adbaabb1697561226aa3c868ce53bcaae209eabab2375b370121dbc02cd1e964"} Jul 27 17:32:38.341071 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:38.340662 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f76bfddc6-7tgpn" Jul 27 17:32:38.341071 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:38.340705 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f76bfddc6-7tgpn" event={"ID":"f34392e3-37f6-4c25-85fd-f3a159873d76","Type":"ContainerDied","Data":"c54344b109e17aef96a681ff48935dabd8042a40f4199b3727a5b9d4f79b9d58"} Jul 27 17:32:38.341071 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:38.340727 2570 scope.go:117] "RemoveContainer" containerID="adbaabb1697561226aa3c868ce53bcaae209eabab2375b370121dbc02cd1e964" Jul 27 17:32:38.348962 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:38.348945 2570 scope.go:117] "RemoveContainer" containerID="adbaabb1697561226aa3c868ce53bcaae209eabab2375b370121dbc02cd1e964" Jul 27 17:32:38.349211 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:32:38.349193 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adbaabb1697561226aa3c868ce53bcaae209eabab2375b370121dbc02cd1e964\": container with ID starting with adbaabb1697561226aa3c868ce53bcaae209eabab2375b370121dbc02cd1e964 not found: ID does not exist" containerID="adbaabb1697561226aa3c868ce53bcaae209eabab2375b370121dbc02cd1e964" Jul 27 17:32:38.349260 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:38.349220 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adbaabb1697561226aa3c868ce53bcaae209eabab2375b370121dbc02cd1e964"} err="failed to get container status \"adbaabb1697561226aa3c868ce53bcaae209eabab2375b370121dbc02cd1e964\": rpc error: code = NotFound desc = could not find container \"adbaabb1697561226aa3c868ce53bcaae209eabab2375b370121dbc02cd1e964\": container with ID starting with adbaabb1697561226aa3c868ce53bcaae209eabab2375b370121dbc02cd1e964 not found: ID does not exist" Jul 27 17:32:38.360866 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:38.360838 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f76bfddc6-7tgpn"] Jul 27 17:32:38.364373 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:38.364352 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6f76bfddc6-7tgpn"] Jul 27 17:32:39.128427 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:32:39.128393 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f34392e3-37f6-4c25-85fd-f3a159873d76" path="/var/lib/kubelet/pods/f34392e3-37f6-4c25-85fd-f3a159873d76/volumes" Jul 27 17:33:00.945963 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:33:00.945926 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-549dcb577f-p5kw6"] Jul 27 17:33:00.946379 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:33:00.946231 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f34392e3-37f6-4c25-85fd-f3a159873d76" containerName="console" Jul 27 17:33:00.946379 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:33:00.946242 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="f34392e3-37f6-4c25-85fd-f3a159873d76" containerName="console" Jul 27 17:33:00.946379 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:33:00.946289 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="f34392e3-37f6-4c25-85fd-f3a159873d76" containerName="console" Jul 27 17:33:00.949437 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:33:00.949415 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-549dcb577f-p5kw6" Jul 27 17:33:00.951677 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:33:00.951657 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 27 17:33:00.951677 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:33:00.951658 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 27 17:33:00.952256 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:33:00.952236 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 27 17:33:00.952348 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:33:00.952274 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 27 17:33:00.952348 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:33:00.952337 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 27 17:33:00.952440 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:33:00.952350 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 27 17:33:00.952583 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:33:00.952568 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 27 17:33:00.960082 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:33:00.960061 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-549dcb577f-p5kw6"] Jul 27 17:33:00.975969 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:33:00.975946 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsjcj\" (UniqueName: \"kubernetes.io/projected/0da321d0-9819-4f79-9d5c-98817d42fd5d-kube-api-access-zsjcj\") pod \"cluster-proxy-proxy-agent-549dcb577f-p5kw6\" (UID: \"0da321d0-9819-4f79-9d5c-98817d42fd5d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-549dcb577f-p5kw6" Jul 27 17:33:00.976077 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:33:00.975993 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0da321d0-9819-4f79-9d5c-98817d42fd5d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-549dcb577f-p5kw6\" (UID: \"0da321d0-9819-4f79-9d5c-98817d42fd5d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-549dcb577f-p5kw6" Jul 27 17:33:00.976077 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:33:00.976049 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/0da321d0-9819-4f79-9d5c-98817d42fd5d-additional-service-ca\") pod \"cluster-proxy-proxy-agent-549dcb577f-p5kw6\" (UID: \"0da321d0-9819-4f79-9d5c-98817d42fd5d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-549dcb577f-p5kw6" Jul 27 17:33:00.976155 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:33:00.976094 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0da321d0-9819-4f79-9d5c-98817d42fd5d-hub\") pod \"cluster-proxy-proxy-agent-549dcb577f-p5kw6\" (UID: \"0da321d0-9819-4f79-9d5c-98817d42fd5d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-549dcb577f-p5kw6" Jul 27 17:33:00.976155 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:33:00.976120 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0da321d0-9819-4f79-9d5c-98817d42fd5d-ca\") pod \"cluster-proxy-proxy-agent-549dcb577f-p5kw6\" (UID: \"0da321d0-9819-4f79-9d5c-98817d42fd5d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-549dcb577f-p5kw6" Jul 27 17:33:00.976155 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:33:00.976143 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/0da321d0-9819-4f79-9d5c-98817d42fd5d-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-549dcb577f-p5kw6\" (UID: \"0da321d0-9819-4f79-9d5c-98817d42fd5d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-549dcb577f-p5kw6" Jul 27 17:33:01.077166 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:33:01.077138 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0da321d0-9819-4f79-9d5c-98817d42fd5d-hub\") pod \"cluster-proxy-proxy-agent-549dcb577f-p5kw6\" (UID: \"0da321d0-9819-4f79-9d5c-98817d42fd5d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-549dcb577f-p5kw6" Jul 27 17:33:01.077166 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:33:01.077168 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0da321d0-9819-4f79-9d5c-98817d42fd5d-ca\") pod \"cluster-proxy-proxy-agent-549dcb577f-p5kw6\" (UID: \"0da321d0-9819-4f79-9d5c-98817d42fd5d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-549dcb577f-p5kw6" Jul 27 17:33:01.077369 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:33:01.077192 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/0da321d0-9819-4f79-9d5c-98817d42fd5d-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-549dcb577f-p5kw6\" (UID: \"0da321d0-9819-4f79-9d5c-98817d42fd5d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-549dcb577f-p5kw6" Jul 27 17:33:01.077369 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:33:01.077224 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsjcj\" (UniqueName: \"kubernetes.io/projected/0da321d0-9819-4f79-9d5c-98817d42fd5d-kube-api-access-zsjcj\") pod \"cluster-proxy-proxy-agent-549dcb577f-p5kw6\" (UID: \"0da321d0-9819-4f79-9d5c-98817d42fd5d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-549dcb577f-p5kw6" Jul 27 17:33:01.077369 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:33:01.077270 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0da321d0-9819-4f79-9d5c-98817d42fd5d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-549dcb577f-p5kw6\" (UID: \"0da321d0-9819-4f79-9d5c-98817d42fd5d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-549dcb577f-p5kw6" Jul 27 17:33:01.077369 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:33:01.077303 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/0da321d0-9819-4f79-9d5c-98817d42fd5d-additional-service-ca\") pod \"cluster-proxy-proxy-agent-549dcb577f-p5kw6\" (UID: \"0da321d0-9819-4f79-9d5c-98817d42fd5d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-549dcb577f-p5kw6" Jul 27 17:33:01.077971 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:33:01.077946 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/0da321d0-9819-4f79-9d5c-98817d42fd5d-additional-service-ca\") pod \"cluster-proxy-proxy-agent-549dcb577f-p5kw6\" (UID: \"0da321d0-9819-4f79-9d5c-98817d42fd5d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-549dcb577f-p5kw6" Jul 27 17:33:01.079865 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:33:01.079834 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/0da321d0-9819-4f79-9d5c-98817d42fd5d-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-549dcb577f-p5kw6\" (UID: \"0da321d0-9819-4f79-9d5c-98817d42fd5d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-549dcb577f-p5kw6" Jul 27 17:33:01.079956 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:33:01.079886 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0da321d0-9819-4f79-9d5c-98817d42fd5d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-549dcb577f-p5kw6\" (UID: \"0da321d0-9819-4f79-9d5c-98817d42fd5d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-549dcb577f-p5kw6" Jul 27 17:33:01.079956 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:33:01.079909 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0da321d0-9819-4f79-9d5c-98817d42fd5d-ca\") pod \"cluster-proxy-proxy-agent-549dcb577f-p5kw6\" (UID: \"0da321d0-9819-4f79-9d5c-98817d42fd5d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-549dcb577f-p5kw6" Jul 27 17:33:01.080156 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:33:01.080135 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0da321d0-9819-4f79-9d5c-98817d42fd5d-hub\") pod \"cluster-proxy-proxy-agent-549dcb577f-p5kw6\" (UID: \"0da321d0-9819-4f79-9d5c-98817d42fd5d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-549dcb577f-p5kw6" Jul 27 17:33:01.085053 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:33:01.085032 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsjcj\" (UniqueName: \"kubernetes.io/projected/0da321d0-9819-4f79-9d5c-98817d42fd5d-kube-api-access-zsjcj\") pod \"cluster-proxy-proxy-agent-549dcb577f-p5kw6\" (UID: \"0da321d0-9819-4f79-9d5c-98817d42fd5d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-549dcb577f-p5kw6" Jul 27 17:33:01.263220 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:33:01.263192 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-549dcb577f-p5kw6" Jul 27 17:33:01.392462 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:33:01.392439 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-549dcb577f-p5kw6"] Jul 27 17:33:01.394559 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:33:01.394506 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0da321d0_9819_4f79_9d5c_98817d42fd5d.slice/crio-b40b23ff97cfe5fa9e56264516b93113793213e55b2f9d2896f60f0f12de7cdc WatchSource:0}: Error finding container b40b23ff97cfe5fa9e56264516b93113793213e55b2f9d2896f60f0f12de7cdc: Status 404 returned error can't find the container with id b40b23ff97cfe5fa9e56264516b93113793213e55b2f9d2896f60f0f12de7cdc Jul 27 17:33:01.430968 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:33:01.430939 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-549dcb577f-p5kw6" event={"ID":"0da321d0-9819-4f79-9d5c-98817d42fd5d","Type":"ContainerStarted","Data":"b40b23ff97cfe5fa9e56264516b93113793213e55b2f9d2896f60f0f12de7cdc"} Jul 27 17:33:02.955841 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:33:02.955805 2570 kubelet.go:1648] "Image garbage collection succeeded" Jul 27 17:33:05.449972 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:33:05.449945 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-549dcb577f-p5kw6" event={"ID":"0da321d0-9819-4f79-9d5c-98817d42fd5d","Type":"ContainerStarted","Data":"b0c0724d7ce30de7705f6b3a756f5a112c320835c66627eb98f2b58e9bfbe244"} Jul 27 17:33:05.450285 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:33:05.449986 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-549dcb577f-p5kw6" event={"ID":"0da321d0-9819-4f79-9d5c-98817d42fd5d","Type":"ContainerStarted","Data":"9d8ed2b9dbf77dad7b4c159d8295440e891b05993c52428dd02f3c9cac83b4bf"} Jul 27 17:33:06.455853 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:33:06.455817 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-549dcb577f-p5kw6" event={"ID":"0da321d0-9819-4f79-9d5c-98817d42fd5d","Type":"ContainerStarted","Data":"2de89d03d7eb0ba9c15aead079264dbed8f7df30e423e3f24de268a1b320f824"} Jul 27 17:33:06.478154 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:33:06.478103 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-549dcb577f-p5kw6" podStartSLOduration=2.6045725600000003 podStartE2EDuration="6.478088112s" totalImagesPullingTime="3.873515552s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:33:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:33:06.476200825 +0000 UTC m=+303.917415799" watchObservedRunningTime="2026-07-27 17:33:06.478088112 +0000 UTC m=+303.919303085" Jul 27 17:35:37.783135 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:35:37.783062 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-84fc959947-q2kzj"] Jul 27 17:35:37.786433 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:35:37.786407 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-84fc959947-q2kzj" Jul 27 17:35:37.788833 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:35:37.788801 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\"" Jul 27 17:35:37.789540 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:35:37.789496 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-bnfx6\"" Jul 27 17:35:37.789643 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:35:37.789546 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" Jul 27 17:35:37.789643 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:35:37.789633 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" Jul 27 17:35:37.796456 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:35:37.796430 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-84fc959947-q2kzj"] Jul 27 17:35:37.906908 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:35:37.906870 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e8295418-67ce-4ee0-87f0-189cd02be6d3-cert\") pod \"llmisvc-controller-manager-84fc959947-q2kzj\" (UID: \"e8295418-67ce-4ee0-87f0-189cd02be6d3\") " pod="kserve/llmisvc-controller-manager-84fc959947-q2kzj" Jul 27 17:35:37.907083 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:35:37.906942 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2sdx\" (UniqueName: \"kubernetes.io/projected/e8295418-67ce-4ee0-87f0-189cd02be6d3-kube-api-access-k2sdx\") pod \"llmisvc-controller-manager-84fc959947-q2kzj\" (UID: \"e8295418-67ce-4ee0-87f0-189cd02be6d3\") " pod="kserve/llmisvc-controller-manager-84fc959947-q2kzj" Jul 27 17:35:38.007595 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:35:38.007562 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e8295418-67ce-4ee0-87f0-189cd02be6d3-cert\") pod \"llmisvc-controller-manager-84fc959947-q2kzj\" (UID: \"e8295418-67ce-4ee0-87f0-189cd02be6d3\") " pod="kserve/llmisvc-controller-manager-84fc959947-q2kzj" Jul 27 17:35:38.007784 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:35:38.007636 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2sdx\" (UniqueName: \"kubernetes.io/projected/e8295418-67ce-4ee0-87f0-189cd02be6d3-kube-api-access-k2sdx\") pod \"llmisvc-controller-manager-84fc959947-q2kzj\" (UID: \"e8295418-67ce-4ee0-87f0-189cd02be6d3\") " pod="kserve/llmisvc-controller-manager-84fc959947-q2kzj" Jul 27 17:35:38.010124 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:35:38.010094 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e8295418-67ce-4ee0-87f0-189cd02be6d3-cert\") pod \"llmisvc-controller-manager-84fc959947-q2kzj\" (UID: \"e8295418-67ce-4ee0-87f0-189cd02be6d3\") " pod="kserve/llmisvc-controller-manager-84fc959947-q2kzj" Jul 27 17:35:38.017775 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:35:38.017742 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2sdx\" (UniqueName: \"kubernetes.io/projected/e8295418-67ce-4ee0-87f0-189cd02be6d3-kube-api-access-k2sdx\") pod \"llmisvc-controller-manager-84fc959947-q2kzj\" (UID: \"e8295418-67ce-4ee0-87f0-189cd02be6d3\") " pod="kserve/llmisvc-controller-manager-84fc959947-q2kzj" Jul 27 17:35:38.097341 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:35:38.097253 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-84fc959947-q2kzj" Jul 27 17:35:38.224417 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:35:38.224386 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-84fc959947-q2kzj"] Jul 27 17:35:38.227949 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:35:38.227923 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode8295418_67ce_4ee0_87f0_189cd02be6d3.slice/crio-fc04e3b4566ad9d75158488a80f104d8f15b0337e58fbd92e5c3288a9348bbbc WatchSource:0}: Error finding container fc04e3b4566ad9d75158488a80f104d8f15b0337e58fbd92e5c3288a9348bbbc: Status 404 returned error can't find the container with id fc04e3b4566ad9d75158488a80f104d8f15b0337e58fbd92e5c3288a9348bbbc Jul 27 17:35:38.229323 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:35:38.229299 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 27 17:35:39.019069 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:35:39.019033 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-84fc959947-q2kzj" event={"ID":"e8295418-67ce-4ee0-87f0-189cd02be6d3","Type":"ContainerStarted","Data":"fc04e3b4566ad9d75158488a80f104d8f15b0337e58fbd92e5c3288a9348bbbc"} Jul 27 17:35:41.030691 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:35:41.030661 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-84fc959947-q2kzj" event={"ID":"e8295418-67ce-4ee0-87f0-189cd02be6d3","Type":"ContainerStarted","Data":"0b563db9a23cf7a8281c32aef3447fec15be93b9eb6ba937b5c566410aefa01d"} Jul 27 17:35:41.031062 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:35:41.030783 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-84fc959947-q2kzj" Jul 27 17:35:41.048937 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:35:41.048831 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-84fc959947-q2kzj" podStartSLOduration=1.494978235 podStartE2EDuration="4.048808207s" totalImagesPullingTime="2.553829972s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:35:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:35:41.048740328 +0000 UTC m=+458.489955294" watchObservedRunningTime="2026-07-27 17:35:41.048808207 +0000 UTC m=+458.490023181" Jul 27 17:36:12.038103 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:36:12.038074 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-84fc959947-q2kzj" Jul 27 17:36:12.408049 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:36:12.407961 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66b76948bc-8ww6t"] Jul 27 17:36:37.427102 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:36:37.427046 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-66b76948bc-8ww6t" podUID="fb6cb5d1-d726-46e3-87af-d83e6af93452" containerName="console" containerID="cri-o://9fc254ed474d6479d2ef3792cdad1f34fa57febb9812d7c55ad63c9acc970f9b" gracePeriod=15 Jul 27 17:36:37.679846 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:36:37.679790 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66b76948bc-8ww6t" Jul 27 17:36:37.822723 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:36:37.822681 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb6cb5d1-d726-46e3-87af-d83e6af93452-trusted-ca-bundle\") pod \"fb6cb5d1-d726-46e3-87af-d83e6af93452\" (UID: \"fb6cb5d1-d726-46e3-87af-d83e6af93452\") " Jul 27 17:36:37.822915 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:36:37.822733 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fb6cb5d1-d726-46e3-87af-d83e6af93452-service-ca\") pod \"fb6cb5d1-d726-46e3-87af-d83e6af93452\" (UID: \"fb6cb5d1-d726-46e3-87af-d83e6af93452\") " Jul 27 17:36:37.822915 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:36:37.822782 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vmch5\" (UniqueName: \"kubernetes.io/projected/fb6cb5d1-d726-46e3-87af-d83e6af93452-kube-api-access-vmch5\") pod \"fb6cb5d1-d726-46e3-87af-d83e6af93452\" (UID: \"fb6cb5d1-d726-46e3-87af-d83e6af93452\") " Jul 27 17:36:37.822915 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:36:37.822805 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fb6cb5d1-d726-46e3-87af-d83e6af93452-oauth-serving-cert\") pod \"fb6cb5d1-d726-46e3-87af-d83e6af93452\" (UID: \"fb6cb5d1-d726-46e3-87af-d83e6af93452\") " Jul 27 17:36:37.823087 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:36:37.822972 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb6cb5d1-d726-46e3-87af-d83e6af93452-console-serving-cert\") pod \"fb6cb5d1-d726-46e3-87af-d83e6af93452\" (UID: \"fb6cb5d1-d726-46e3-87af-d83e6af93452\") " Jul 27 17:36:37.823087 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:36:37.823036 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fb6cb5d1-d726-46e3-87af-d83e6af93452-console-config\") pod \"fb6cb5d1-d726-46e3-87af-d83e6af93452\" (UID: \"fb6cb5d1-d726-46e3-87af-d83e6af93452\") " Jul 27 17:36:37.823087 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:36:37.823070 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fb6cb5d1-d726-46e3-87af-d83e6af93452-console-oauth-config\") pod \"fb6cb5d1-d726-46e3-87af-d83e6af93452\" (UID: \"fb6cb5d1-d726-46e3-87af-d83e6af93452\") " Jul 27 17:36:37.823238 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:36:37.823090 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb6cb5d1-d726-46e3-87af-d83e6af93452-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "fb6cb5d1-d726-46e3-87af-d83e6af93452" (UID: "fb6cb5d1-d726-46e3-87af-d83e6af93452"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:36:37.823238 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:36:37.823218 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb6cb5d1-d726-46e3-87af-d83e6af93452-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "fb6cb5d1-d726-46e3-87af-d83e6af93452" (UID: "fb6cb5d1-d726-46e3-87af-d83e6af93452"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:36:37.823342 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:36:37.823239 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb6cb5d1-d726-46e3-87af-d83e6af93452-service-ca" (OuterVolumeSpecName: "service-ca") pod "fb6cb5d1-d726-46e3-87af-d83e6af93452" (UID: "fb6cb5d1-d726-46e3-87af-d83e6af93452"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:36:37.823342 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:36:37.823315 2570 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fb6cb5d1-d726-46e3-87af-d83e6af93452-service-ca\") on node \"ip-10-0-140-185.ec2.internal\" DevicePath \"\"" Jul 27 17:36:37.823342 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:36:37.823334 2570 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fb6cb5d1-d726-46e3-87af-d83e6af93452-oauth-serving-cert\") on node \"ip-10-0-140-185.ec2.internal\" DevicePath \"\"" Jul 27 17:36:37.823444 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:36:37.823347 2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb6cb5d1-d726-46e3-87af-d83e6af93452-trusted-ca-bundle\") on node \"ip-10-0-140-185.ec2.internal\" DevicePath \"\"" Jul 27 17:36:37.823444 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:36:37.823354 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb6cb5d1-d726-46e3-87af-d83e6af93452-console-config" (OuterVolumeSpecName: "console-config") pod "fb6cb5d1-d726-46e3-87af-d83e6af93452" (UID: "fb6cb5d1-d726-46e3-87af-d83e6af93452"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:36:37.824946 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:36:37.824923 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb6cb5d1-d726-46e3-87af-d83e6af93452-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "fb6cb5d1-d726-46e3-87af-d83e6af93452" (UID: "fb6cb5d1-d726-46e3-87af-d83e6af93452"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 17:36:37.825071 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:36:37.825056 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb6cb5d1-d726-46e3-87af-d83e6af93452-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "fb6cb5d1-d726-46e3-87af-d83e6af93452" (UID: "fb6cb5d1-d726-46e3-87af-d83e6af93452"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 17:36:37.825181 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:36:37.825162 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb6cb5d1-d726-46e3-87af-d83e6af93452-kube-api-access-vmch5" (OuterVolumeSpecName: "kube-api-access-vmch5") pod "fb6cb5d1-d726-46e3-87af-d83e6af93452" (UID: "fb6cb5d1-d726-46e3-87af-d83e6af93452"). InnerVolumeSpecName "kube-api-access-vmch5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 17:36:37.923752 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:36:37.923714 2570 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fb6cb5d1-d726-46e3-87af-d83e6af93452-console-config\") on node \"ip-10-0-140-185.ec2.internal\" DevicePath \"\"" Jul 27 17:36:37.923752 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:36:37.923744 2570 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fb6cb5d1-d726-46e3-87af-d83e6af93452-console-oauth-config\") on node \"ip-10-0-140-185.ec2.internal\" DevicePath \"\"" Jul 27 17:36:37.923752 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:36:37.923757 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmch5\" (UniqueName: \"kubernetes.io/projected/fb6cb5d1-d726-46e3-87af-d83e6af93452-kube-api-access-vmch5\") on node \"ip-10-0-140-185.ec2.internal\" DevicePath \"\"" Jul 27 17:36:37.923978 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:36:37.923769 2570 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb6cb5d1-d726-46e3-87af-d83e6af93452-console-serving-cert\") on node \"ip-10-0-140-185.ec2.internal\" DevicePath \"\"" Jul 27 17:36:38.251247 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:36:38.251213 2570 generic.go:356] "Generic (PLEG): container finished" podID="fb6cb5d1-d726-46e3-87af-d83e6af93452" containerID="9fc254ed474d6479d2ef3792cdad1f34fa57febb9812d7c55ad63c9acc970f9b" exitCode=0 Jul 27 17:36:38.251430 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:36:38.251259 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66b76948bc-8ww6t" Jul 27 17:36:38.251430 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:36:38.251300 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66b76948bc-8ww6t" event={"ID":"fb6cb5d1-d726-46e3-87af-d83e6af93452","Type":"ContainerDied","Data":"9fc254ed474d6479d2ef3792cdad1f34fa57febb9812d7c55ad63c9acc970f9b"} Jul 27 17:36:38.251430 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:36:38.251353 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66b76948bc-8ww6t" event={"ID":"fb6cb5d1-d726-46e3-87af-d83e6af93452","Type":"ContainerDied","Data":"137dd298a7b484bd0bfdbe35c098935e1cce400450de465ae219bf3fd4b55e74"} Jul 27 17:36:38.251430 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:36:38.251380 2570 scope.go:117] "RemoveContainer" containerID="9fc254ed474d6479d2ef3792cdad1f34fa57febb9812d7c55ad63c9acc970f9b" Jul 27 17:36:38.260173 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:36:38.260156 2570 scope.go:117] "RemoveContainer" containerID="9fc254ed474d6479d2ef3792cdad1f34fa57febb9812d7c55ad63c9acc970f9b" Jul 27 17:36:38.260429 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:36:38.260408 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fc254ed474d6479d2ef3792cdad1f34fa57febb9812d7c55ad63c9acc970f9b\": container with ID starting with 9fc254ed474d6479d2ef3792cdad1f34fa57febb9812d7c55ad63c9acc970f9b not found: ID does not exist" containerID="9fc254ed474d6479d2ef3792cdad1f34fa57febb9812d7c55ad63c9acc970f9b" Jul 27 17:36:38.260473 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:36:38.260438 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fc254ed474d6479d2ef3792cdad1f34fa57febb9812d7c55ad63c9acc970f9b"} err="failed to get container status \"9fc254ed474d6479d2ef3792cdad1f34fa57febb9812d7c55ad63c9acc970f9b\": rpc error: code = NotFound desc = could not find container \"9fc254ed474d6479d2ef3792cdad1f34fa57febb9812d7c55ad63c9acc970f9b\": container with ID starting with 9fc254ed474d6479d2ef3792cdad1f34fa57febb9812d7c55ad63c9acc970f9b not found: ID does not exist" Jul 27 17:36:38.271120 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:36:38.271098 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66b76948bc-8ww6t"] Jul 27 17:36:38.274501 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:36:38.274481 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-66b76948bc-8ww6t"] Jul 27 17:36:39.128603 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:36:39.128563 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb6cb5d1-d726-46e3-87af-d83e6af93452" path="/var/lib/kubelet/pods/fb6cb5d1-d726-46e3-87af-d83e6af93452/volumes" Jul 27 17:36:53.418281 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:36:53.418248 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-z6jd2"] Jul 27 17:36:53.418820 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:36:53.418703 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb6cb5d1-d726-46e3-87af-d83e6af93452" containerName="console" Jul 27 17:36:53.418820 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:36:53.418721 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb6cb5d1-d726-46e3-87af-d83e6af93452" containerName="console" Jul 27 17:36:53.418820 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:36:53.418818 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb6cb5d1-d726-46e3-87af-d83e6af93452" containerName="console" Jul 27 17:36:53.423252 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:36:53.423232 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-z6jd2" Jul 27 17:36:53.425296 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:36:53.425276 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\"" Jul 27 17:36:53.425463 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:36:53.425400 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-l46pn\"" Jul 27 17:36:53.430811 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:36:53.430789 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-z6jd2"] Jul 27 17:36:53.451784 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:36:53.451751 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c56b11d-42a2-4a68-900a-704bf8d4bb8c-tls-certs\") pod \"model-serving-api-7b954ffc95-z6jd2\" (UID: \"1c56b11d-42a2-4a68-900a-704bf8d4bb8c\") " pod="kserve/model-serving-api-7b954ffc95-z6jd2" Jul 27 17:36:53.451784 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:36:53.451786 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hhn4\" (UniqueName: \"kubernetes.io/projected/1c56b11d-42a2-4a68-900a-704bf8d4bb8c-kube-api-access-8hhn4\") pod \"model-serving-api-7b954ffc95-z6jd2\" (UID: \"1c56b11d-42a2-4a68-900a-704bf8d4bb8c\") " pod="kserve/model-serving-api-7b954ffc95-z6jd2" Jul 27 17:36:53.552397 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:36:53.552354 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c56b11d-42a2-4a68-900a-704bf8d4bb8c-tls-certs\") pod \"model-serving-api-7b954ffc95-z6jd2\" (UID: \"1c56b11d-42a2-4a68-900a-704bf8d4bb8c\") " pod="kserve/model-serving-api-7b954ffc95-z6jd2" Jul 27 17:36:53.552397 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:36:53.552399 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hhn4\" (UniqueName: \"kubernetes.io/projected/1c56b11d-42a2-4a68-900a-704bf8d4bb8c-kube-api-access-8hhn4\") pod \"model-serving-api-7b954ffc95-z6jd2\" (UID: \"1c56b11d-42a2-4a68-900a-704bf8d4bb8c\") " pod="kserve/model-serving-api-7b954ffc95-z6jd2" Jul 27 17:36:53.552692 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:36:53.552514 2570 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 27 17:36:53.552692 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:36:53.552612 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c56b11d-42a2-4a68-900a-704bf8d4bb8c-tls-certs podName:1c56b11d-42a2-4a68-900a-704bf8d4bb8c nodeName:}" failed. No retries permitted until 2026-07-27 17:36:54.05259144 +0000 UTC m=+531.493806396 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/1c56b11d-42a2-4a68-900a-704bf8d4bb8c-tls-certs") pod "model-serving-api-7b954ffc95-z6jd2" (UID: "1c56b11d-42a2-4a68-900a-704bf8d4bb8c") : secret "model-serving-api-tls" not found Jul 27 17:36:53.560480 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:36:53.560453 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hhn4\" (UniqueName: \"kubernetes.io/projected/1c56b11d-42a2-4a68-900a-704bf8d4bb8c-kube-api-access-8hhn4\") pod \"model-serving-api-7b954ffc95-z6jd2\" (UID: \"1c56b11d-42a2-4a68-900a-704bf8d4bb8c\") " pod="kserve/model-serving-api-7b954ffc95-z6jd2" Jul 27 17:36:54.055807 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:36:54.055763 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c56b11d-42a2-4a68-900a-704bf8d4bb8c-tls-certs\") pod \"model-serving-api-7b954ffc95-z6jd2\" (UID: \"1c56b11d-42a2-4a68-900a-704bf8d4bb8c\") " pod="kserve/model-serving-api-7b954ffc95-z6jd2" Jul 27 17:36:54.058492 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:36:54.058467 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c56b11d-42a2-4a68-900a-704bf8d4bb8c-tls-certs\") pod \"model-serving-api-7b954ffc95-z6jd2\" (UID: \"1c56b11d-42a2-4a68-900a-704bf8d4bb8c\") " pod="kserve/model-serving-api-7b954ffc95-z6jd2" Jul 27 17:36:54.334531 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:36:54.334428 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-z6jd2" Jul 27 17:36:54.464259 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:36:54.464234 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-z6jd2"] Jul 27 17:36:54.466137 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:36:54.466108 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c56b11d_42a2_4a68_900a_704bf8d4bb8c.slice/crio-5982b3291f80636c8f1ea8809af0b1b484101413cacb6ae311fc76f37b27a083 WatchSource:0}: Error finding container 5982b3291f80636c8f1ea8809af0b1b484101413cacb6ae311fc76f37b27a083: Status 404 returned error can't find the container with id 5982b3291f80636c8f1ea8809af0b1b484101413cacb6ae311fc76f37b27a083 Jul 27 17:36:55.318880 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:36:55.318824 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-z6jd2" event={"ID":"1c56b11d-42a2-4a68-900a-704bf8d4bb8c","Type":"ContainerStarted","Data":"5982b3291f80636c8f1ea8809af0b1b484101413cacb6ae311fc76f37b27a083"} Jul 27 17:36:57.328593 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:36:57.328555 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-z6jd2" event={"ID":"1c56b11d-42a2-4a68-900a-704bf8d4bb8c","Type":"ContainerStarted","Data":"4d3027602a5f054a0495f66390c73210e1efaf7af4415d87b9a73f5c1fd29dcd"} Jul 27 17:36:57.328963 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:36:57.328680 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-z6jd2" Jul 27 17:36:57.351203 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:36:57.351157 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-z6jd2" podStartSLOduration=2.012228642 podStartE2EDuration="4.351140629s" totalImagesPullingTime="2.338911987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:36:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:36:57.349740788 +0000 UTC m=+534.790955776" watchObservedRunningTime="2026-07-27 17:36:57.351140629 +0000 UTC m=+534.792355656" Jul 27 17:37:08.336815 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:37:08.336786 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-z6jd2" Jul 27 17:40:00.468015 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:40:00.467979 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-24ce3-86f4df8849-ngk5d"] Jul 27 17:40:00.471279 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:40:00.471261 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-24ce3-86f4df8849-ngk5d" Jul 27 17:40:00.473389 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:40:00.473366 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-x6c7x\"" Jul 27 17:40:00.473389 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:40:00.473381 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-24ce3-serving-cert\"" Jul 27 17:40:00.473560 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:40:00.473366 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-24ce3-kube-rbac-proxy-sar-config\"" Jul 27 17:40:00.473676 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:40:00.473653 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 27 17:40:00.481022 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:40:00.481001 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-24ce3-86f4df8849-ngk5d"] Jul 27 17:40:00.596875 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:40:00.596839 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2f32782-f266-42c7-bf94-aeb3a1c097b1-openshift-service-ca-bundle\") pod \"switch-graph-24ce3-86f4df8849-ngk5d\" (UID: \"e2f32782-f266-42c7-bf94-aeb3a1c097b1\") " pod="kserve-ci-e2e-test/switch-graph-24ce3-86f4df8849-ngk5d" Jul 27 17:40:00.596875 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:40:00.596876 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2f32782-f266-42c7-bf94-aeb3a1c097b1-proxy-tls\") pod \"switch-graph-24ce3-86f4df8849-ngk5d\" (UID: \"e2f32782-f266-42c7-bf94-aeb3a1c097b1\") " pod="kserve-ci-e2e-test/switch-graph-24ce3-86f4df8849-ngk5d" Jul 27 17:40:00.697376 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:40:00.697322 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2f32782-f266-42c7-bf94-aeb3a1c097b1-openshift-service-ca-bundle\") pod \"switch-graph-24ce3-86f4df8849-ngk5d\" (UID: \"e2f32782-f266-42c7-bf94-aeb3a1c097b1\") " pod="kserve-ci-e2e-test/switch-graph-24ce3-86f4df8849-ngk5d" Jul 27 17:40:00.697376 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:40:00.697384 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2f32782-f266-42c7-bf94-aeb3a1c097b1-proxy-tls\") pod \"switch-graph-24ce3-86f4df8849-ngk5d\" (UID: \"e2f32782-f266-42c7-bf94-aeb3a1c097b1\") " pod="kserve-ci-e2e-test/switch-graph-24ce3-86f4df8849-ngk5d" Jul 27 17:40:00.697696 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:40:00.697555 2570 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-24ce3-serving-cert: secret "switch-graph-24ce3-serving-cert" not found Jul 27 17:40:00.697696 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:40:00.697638 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2f32782-f266-42c7-bf94-aeb3a1c097b1-proxy-tls podName:e2f32782-f266-42c7-bf94-aeb3a1c097b1 nodeName:}" failed. No retries permitted until 2026-07-27 17:40:01.197616447 +0000 UTC m=+718.638831415 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e2f32782-f266-42c7-bf94-aeb3a1c097b1-proxy-tls") pod "switch-graph-24ce3-86f4df8849-ngk5d" (UID: "e2f32782-f266-42c7-bf94-aeb3a1c097b1") : secret "switch-graph-24ce3-serving-cert" not found Jul 27 17:40:00.698087 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:40:00.698066 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2f32782-f266-42c7-bf94-aeb3a1c097b1-openshift-service-ca-bundle\") pod \"switch-graph-24ce3-86f4df8849-ngk5d\" (UID: \"e2f32782-f266-42c7-bf94-aeb3a1c097b1\") " pod="kserve-ci-e2e-test/switch-graph-24ce3-86f4df8849-ngk5d" Jul 27 17:40:01.204206 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:40:01.204171 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2f32782-f266-42c7-bf94-aeb3a1c097b1-proxy-tls\") pod \"switch-graph-24ce3-86f4df8849-ngk5d\" (UID: \"e2f32782-f266-42c7-bf94-aeb3a1c097b1\") " pod="kserve-ci-e2e-test/switch-graph-24ce3-86f4df8849-ngk5d" Jul 27 17:40:01.206735 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:40:01.206709 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2f32782-f266-42c7-bf94-aeb3a1c097b1-proxy-tls\") pod \"switch-graph-24ce3-86f4df8849-ngk5d\" (UID: \"e2f32782-f266-42c7-bf94-aeb3a1c097b1\") " pod="kserve-ci-e2e-test/switch-graph-24ce3-86f4df8849-ngk5d" Jul 27 17:40:01.381637 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:40:01.381604 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-24ce3-86f4df8849-ngk5d" Jul 27 17:40:01.507318 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:40:01.507210 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-24ce3-86f4df8849-ngk5d"] Jul 27 17:40:01.510155 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:40:01.510128 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2f32782_f266_42c7_bf94_aeb3a1c097b1.slice/crio-406d80b3055ad800a4c53a2c7c87421e9943c49dc1d9f72453625a1258773953 WatchSource:0}: Error finding container 406d80b3055ad800a4c53a2c7c87421e9943c49dc1d9f72453625a1258773953: Status 404 returned error can't find the container with id 406d80b3055ad800a4c53a2c7c87421e9943c49dc1d9f72453625a1258773953 Jul 27 17:40:02.023681 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:40:02.023640 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-24ce3-86f4df8849-ngk5d" event={"ID":"e2f32782-f266-42c7-bf94-aeb3a1c097b1","Type":"ContainerStarted","Data":"406d80b3055ad800a4c53a2c7c87421e9943c49dc1d9f72453625a1258773953"} Jul 27 17:40:04.039364 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:40:04.039326 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-24ce3-86f4df8849-ngk5d" event={"ID":"e2f32782-f266-42c7-bf94-aeb3a1c097b1","Type":"ContainerStarted","Data":"7c145b193fb107dcce054566fcfc33e374d889da9e585beef2147fae9707d80c"} Jul 27 17:40:04.039852 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:40:04.039453 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-24ce3-86f4df8849-ngk5d" Jul 27 17:40:04.054813 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:40:04.054767 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-24ce3-86f4df8849-ngk5d" podStartSLOduration=1.644434706 podStartE2EDuration="4.054751403s" totalImagesPullingTime="2.410316697s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:40:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:40:04.054042807 +0000 UTC m=+721.495257782" watchObservedRunningTime="2026-07-27 17:40:04.054751403 +0000 UTC m=+721.495966385" Jul 27 17:40:10.048788 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:40:10.048760 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-24ce3-86f4df8849-ngk5d" Jul 27 17:40:10.589006 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:40:10.588976 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-24ce3-86f4df8849-ngk5d"] Jul 27 17:40:10.589225 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:40:10.589202 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-24ce3-86f4df8849-ngk5d" podUID="e2f32782-f266-42c7-bf94-aeb3a1c097b1" containerName="switch-graph-24ce3" containerID="cri-o://7c145b193fb107dcce054566fcfc33e374d889da9e585beef2147fae9707d80c" gracePeriod=30 Jul 27 17:40:15.046673 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:40:15.046633 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-24ce3-86f4df8849-ngk5d" podUID="e2f32782-f266-42c7-bf94-aeb3a1c097b1" containerName="switch-graph-24ce3" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 17:40:20.047151 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:40:20.047110 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-24ce3-86f4df8849-ngk5d" podUID="e2f32782-f266-42c7-bf94-aeb3a1c097b1" containerName="switch-graph-24ce3" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 17:40:25.047303 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:40:25.047261 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-24ce3-86f4df8849-ngk5d" podUID="e2f32782-f266-42c7-bf94-aeb3a1c097b1" containerName="switch-graph-24ce3" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 17:40:25.047685 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:40:25.047361 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-24ce3-86f4df8849-ngk5d" Jul 27 17:40:30.047494 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:40:30.047448 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-24ce3-86f4df8849-ngk5d" podUID="e2f32782-f266-42c7-bf94-aeb3a1c097b1" containerName="switch-graph-24ce3" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 17:40:35.046759 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:40:35.046715 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-24ce3-86f4df8849-ngk5d" podUID="e2f32782-f266-42c7-bf94-aeb3a1c097b1" containerName="switch-graph-24ce3" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 17:40:40.047218 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:40:40.047182 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-24ce3-86f4df8849-ngk5d" podUID="e2f32782-f266-42c7-bf94-aeb3a1c097b1" containerName="switch-graph-24ce3" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 17:40:41.182755 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:40:41.182716 2570 generic.go:356] "Generic (PLEG): container finished" podID="e2f32782-f266-42c7-bf94-aeb3a1c097b1" containerID="7c145b193fb107dcce054566fcfc33e374d889da9e585beef2147fae9707d80c" exitCode=0 Jul 27 17:40:41.183125 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:40:41.182780 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-24ce3-86f4df8849-ngk5d" event={"ID":"e2f32782-f266-42c7-bf94-aeb3a1c097b1","Type":"ContainerDied","Data":"7c145b193fb107dcce054566fcfc33e374d889da9e585beef2147fae9707d80c"} Jul 27 17:40:41.238043 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:40:41.238021 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-24ce3-86f4df8849-ngk5d" Jul 27 17:40:41.337607 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:40:41.337572 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2f32782-f266-42c7-bf94-aeb3a1c097b1-proxy-tls\") pod \"e2f32782-f266-42c7-bf94-aeb3a1c097b1\" (UID: \"e2f32782-f266-42c7-bf94-aeb3a1c097b1\") " Jul 27 17:40:41.337820 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:40:41.337646 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2f32782-f266-42c7-bf94-aeb3a1c097b1-openshift-service-ca-bundle\") pod \"e2f32782-f266-42c7-bf94-aeb3a1c097b1\" (UID: \"e2f32782-f266-42c7-bf94-aeb3a1c097b1\") " Jul 27 17:40:41.338043 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:40:41.338014 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2f32782-f266-42c7-bf94-aeb3a1c097b1-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "e2f32782-f266-42c7-bf94-aeb3a1c097b1" (UID: "e2f32782-f266-42c7-bf94-aeb3a1c097b1"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:40:41.339838 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:40:41.339815 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2f32782-f266-42c7-bf94-aeb3a1c097b1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e2f32782-f266-42c7-bf94-aeb3a1c097b1" (UID: "e2f32782-f266-42c7-bf94-aeb3a1c097b1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 17:40:41.438926 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:40:41.438837 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2f32782-f266-42c7-bf94-aeb3a1c097b1-openshift-service-ca-bundle\") on node \"ip-10-0-140-185.ec2.internal\" DevicePath \"\"" Jul 27 17:40:41.438926 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:40:41.438872 2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2f32782-f266-42c7-bf94-aeb3a1c097b1-proxy-tls\") on node \"ip-10-0-140-185.ec2.internal\" DevicePath \"\"" Jul 27 17:40:42.187763 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:40:42.187730 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-24ce3-86f4df8849-ngk5d" Jul 27 17:40:42.187763 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:40:42.187740 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-24ce3-86f4df8849-ngk5d" event={"ID":"e2f32782-f266-42c7-bf94-aeb3a1c097b1","Type":"ContainerDied","Data":"406d80b3055ad800a4c53a2c7c87421e9943c49dc1d9f72453625a1258773953"} Jul 27 17:40:42.188236 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:40:42.187804 2570 scope.go:117] "RemoveContainer" containerID="7c145b193fb107dcce054566fcfc33e374d889da9e585beef2147fae9707d80c" Jul 27 17:40:42.210761 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:40:42.210731 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-24ce3-86f4df8849-ngk5d"] Jul 27 17:40:42.214337 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:40:42.214316 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-24ce3-86f4df8849-ngk5d"] Jul 27 17:40:43.128919 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:40:43.128880 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2f32782-f266-42c7-bf94-aeb3a1c097b1" path="/var/lib/kubelet/pods/e2f32782-f266-42c7-bf94-aeb3a1c097b1/volumes" Jul 27 17:41:00.528288 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:41:00.528252 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-b84f4c8c7-jn5ph"] Jul 27 17:41:00.528708 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:41:00.528637 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2f32782-f266-42c7-bf94-aeb3a1c097b1" containerName="switch-graph-24ce3" Jul 27 17:41:00.528708 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:41:00.528651 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2f32782-f266-42c7-bf94-aeb3a1c097b1" containerName="switch-graph-24ce3" Jul 27 17:41:00.528708 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:41:00.528699 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2f32782-f266-42c7-bf94-aeb3a1c097b1" containerName="switch-graph-24ce3" Jul 27 17:41:00.531481 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:41:00.531462 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-jn5ph" Jul 27 17:41:00.533836 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:41:00.533810 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 27 17:41:00.533952 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:41:00.533820 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-kube-rbac-proxy-sar-config\"" Jul 27 17:41:00.533952 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:41:00.533820 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-serving-cert\"" Jul 27 17:41:00.534075 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:41:00.534028 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-x6c7x\"" Jul 27 17:41:00.541406 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:41:00.541383 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-b84f4c8c7-jn5ph"] Jul 27 17:41:00.609096 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:41:00.609060 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72543746-44c6-417b-a368-8a35cb248743-proxy-tls\") pod \"model-chainer-b84f4c8c7-jn5ph\" (UID: \"72543746-44c6-417b-a368-8a35cb248743\") " pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-jn5ph" Jul 27 17:41:00.609277 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:41:00.609127 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72543746-44c6-417b-a368-8a35cb248743-openshift-service-ca-bundle\") pod \"model-chainer-b84f4c8c7-jn5ph\" (UID: \"72543746-44c6-417b-a368-8a35cb248743\") " pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-jn5ph" Jul 27 17:41:00.710397 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:41:00.710350 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72543746-44c6-417b-a368-8a35cb248743-openshift-service-ca-bundle\") pod \"model-chainer-b84f4c8c7-jn5ph\" (UID: \"72543746-44c6-417b-a368-8a35cb248743\") " pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-jn5ph" Jul 27 17:41:00.710628 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:41:00.710459 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72543746-44c6-417b-a368-8a35cb248743-proxy-tls\") pod \"model-chainer-b84f4c8c7-jn5ph\" (UID: \"72543746-44c6-417b-a368-8a35cb248743\") " pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-jn5ph" Jul 27 17:41:00.710628 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:41:00.710602 2570 secret.go:189] Couldn't get secret kserve-ci-e2e-test/model-chainer-serving-cert: secret "model-chainer-serving-cert" not found Jul 27 17:41:00.710758 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:41:00.710681 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72543746-44c6-417b-a368-8a35cb248743-proxy-tls podName:72543746-44c6-417b-a368-8a35cb248743 nodeName:}" failed. No retries permitted until 2026-07-27 17:41:01.210658512 +0000 UTC m=+778.651873474 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/72543746-44c6-417b-a368-8a35cb248743-proxy-tls") pod "model-chainer-b84f4c8c7-jn5ph" (UID: "72543746-44c6-417b-a368-8a35cb248743") : secret "model-chainer-serving-cert" not found Jul 27 17:41:00.711011 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:41:00.710988 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72543746-44c6-417b-a368-8a35cb248743-openshift-service-ca-bundle\") pod \"model-chainer-b84f4c8c7-jn5ph\" (UID: \"72543746-44c6-417b-a368-8a35cb248743\") " pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-jn5ph" Jul 27 17:41:01.214595 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:41:01.214552 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72543746-44c6-417b-a368-8a35cb248743-proxy-tls\") pod \"model-chainer-b84f4c8c7-jn5ph\" (UID: \"72543746-44c6-417b-a368-8a35cb248743\") " pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-jn5ph" Jul 27 17:41:01.217053 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:41:01.217020 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72543746-44c6-417b-a368-8a35cb248743-proxy-tls\") pod \"model-chainer-b84f4c8c7-jn5ph\" (UID: \"72543746-44c6-417b-a368-8a35cb248743\") " pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-jn5ph" Jul 27 17:41:01.442336 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:41:01.442297 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-jn5ph" Jul 27 17:41:01.568120 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:41:01.568096 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-b84f4c8c7-jn5ph"] Jul 27 17:41:01.570487 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:41:01.570459 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72543746_44c6_417b_a368_8a35cb248743.slice/crio-3baabd7aa5090ff6e3fa8f40326ae830d2dce26bc2a9e90b1737a281b988d748 WatchSource:0}: Error finding container 3baabd7aa5090ff6e3fa8f40326ae830d2dce26bc2a9e90b1737a281b988d748: Status 404 returned error can't find the container with id 3baabd7aa5090ff6e3fa8f40326ae830d2dce26bc2a9e90b1737a281b988d748 Jul 27 17:41:02.264827 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:41:02.264797 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-jn5ph" event={"ID":"72543746-44c6-417b-a368-8a35cb248743","Type":"ContainerStarted","Data":"7fd2d1442f9e41d22c2c80ac89cb4e5e9cc3d8d88a51c16a748903fbc8278990"} Jul 27 17:41:02.265009 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:41:02.264845 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-jn5ph" event={"ID":"72543746-44c6-417b-a368-8a35cb248743","Type":"ContainerStarted","Data":"3baabd7aa5090ff6e3fa8f40326ae830d2dce26bc2a9e90b1737a281b988d748"} Jul 27 17:41:02.265009 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:41:02.264933 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-jn5ph" Jul 27 17:41:02.284046 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:41:02.283991 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-jn5ph" podStartSLOduration=2.28397454 podStartE2EDuration="2.28397454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:41:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:41:02.282891656 +0000 UTC m=+779.724106630" watchObservedRunningTime="2026-07-27 17:41:02.28397454 +0000 UTC m=+779.725189513" Jul 27 17:41:08.275913 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:41:08.275879 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-jn5ph" Jul 27 17:41:10.613083 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:41:10.613050 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-b84f4c8c7-jn5ph"] Jul 27 17:41:10.613559 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:41:10.613250 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-jn5ph" podUID="72543746-44c6-417b-a368-8a35cb248743" containerName="model-chainer" containerID="cri-o://7fd2d1442f9e41d22c2c80ac89cb4e5e9cc3d8d88a51c16a748903fbc8278990" gracePeriod=30 Jul 27 17:41:10.820304 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:41:10.820276 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-78126-67c9695b9f-vxqd4"] Jul 27 17:41:10.824710 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:41:10.824679 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-78126-67c9695b9f-vxqd4" Jul 27 17:41:10.826692 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:41:10.826674 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-78126-serving-cert\"" Jul 27 17:41:10.826692 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:41:10.826681 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-78126-kube-rbac-proxy-sar-config\"" Jul 27 17:41:10.834403 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:41:10.834375 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-78126-67c9695b9f-vxqd4"] Jul 27 17:41:10.995475 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:41:10.995385 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/96d6b210-37d9-4d67-b436-bb1d264dc3df-proxy-tls\") pod \"switch-graph-78126-67c9695b9f-vxqd4\" (UID: \"96d6b210-37d9-4d67-b436-bb1d264dc3df\") " pod="kserve-ci-e2e-test/switch-graph-78126-67c9695b9f-vxqd4" Jul 27 17:41:10.995656 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:41:10.995537 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96d6b210-37d9-4d67-b436-bb1d264dc3df-openshift-service-ca-bundle\") pod \"switch-graph-78126-67c9695b9f-vxqd4\" (UID: \"96d6b210-37d9-4d67-b436-bb1d264dc3df\") " pod="kserve-ci-e2e-test/switch-graph-78126-67c9695b9f-vxqd4" Jul 27 17:41:11.096374 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:41:11.096336 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96d6b210-37d9-4d67-b436-bb1d264dc3df-openshift-service-ca-bundle\") pod \"switch-graph-78126-67c9695b9f-vxqd4\" (UID: \"96d6b210-37d9-4d67-b436-bb1d264dc3df\") " pod="kserve-ci-e2e-test/switch-graph-78126-67c9695b9f-vxqd4" Jul 27 17:41:11.096374 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:41:11.096378 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/96d6b210-37d9-4d67-b436-bb1d264dc3df-proxy-tls\") pod \"switch-graph-78126-67c9695b9f-vxqd4\" (UID: \"96d6b210-37d9-4d67-b436-bb1d264dc3df\") " pod="kserve-ci-e2e-test/switch-graph-78126-67c9695b9f-vxqd4" Jul 27 17:41:11.097015 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:41:11.096987 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96d6b210-37d9-4d67-b436-bb1d264dc3df-openshift-service-ca-bundle\") pod \"switch-graph-78126-67c9695b9f-vxqd4\" (UID: \"96d6b210-37d9-4d67-b436-bb1d264dc3df\") " pod="kserve-ci-e2e-test/switch-graph-78126-67c9695b9f-vxqd4" Jul 27 17:41:11.098826 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:41:11.098797 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/96d6b210-37d9-4d67-b436-bb1d264dc3df-proxy-tls\") pod \"switch-graph-78126-67c9695b9f-vxqd4\" (UID: \"96d6b210-37d9-4d67-b436-bb1d264dc3df\") " pod="kserve-ci-e2e-test/switch-graph-78126-67c9695b9f-vxqd4" Jul 27 17:41:11.137228 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:41:11.137201 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-78126-67c9695b9f-vxqd4" Jul 27 17:41:11.266970 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:41:11.266944 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-78126-67c9695b9f-vxqd4"] Jul 27 17:41:11.269252 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:41:11.269226 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96d6b210_37d9_4d67_b436_bb1d264dc3df.slice/crio-f82db4981c0fbb3bee8371f44a619dbc4e505dd280b5394f763e45c5050fbdd3 WatchSource:0}: Error finding container f82db4981c0fbb3bee8371f44a619dbc4e505dd280b5394f763e45c5050fbdd3: Status 404 returned error can't find the container with id f82db4981c0fbb3bee8371f44a619dbc4e505dd280b5394f763e45c5050fbdd3 Jul 27 17:41:11.301141 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:41:11.301114 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-78126-67c9695b9f-vxqd4" event={"ID":"96d6b210-37d9-4d67-b436-bb1d264dc3df","Type":"ContainerStarted","Data":"f82db4981c0fbb3bee8371f44a619dbc4e505dd280b5394f763e45c5050fbdd3"} Jul 27 17:41:12.312442 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:41:12.312404 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-78126-67c9695b9f-vxqd4" event={"ID":"96d6b210-37d9-4d67-b436-bb1d264dc3df","Type":"ContainerStarted","Data":"575508041b25ad26675c43253be0b823a93773e3f34664d1067dc4ada46785ff"} Jul 27 17:41:12.312945 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:41:12.312572 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-78126-67c9695b9f-vxqd4" Jul 27 17:41:12.329647 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:41:12.329597 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-78126-67c9695b9f-vxqd4" podStartSLOduration=2.329581261 podStartE2EDuration="2.329581261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:41:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:41:12.328725441 +0000 UTC m=+789.769940417" watchObservedRunningTime="2026-07-27 17:41:12.329581261 +0000 UTC m=+789.770796235" Jul 27 17:41:13.274805 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:41:13.274767 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-jn5ph" podUID="72543746-44c6-417b-a368-8a35cb248743" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 17:41:18.274772 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:41:18.274732 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-jn5ph" podUID="72543746-44c6-417b-a368-8a35cb248743" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 17:41:18.321740 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:41:18.321708 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-78126-67c9695b9f-vxqd4" Jul 27 17:41:23.275615 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:41:23.275570 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-jn5ph" podUID="72543746-44c6-417b-a368-8a35cb248743" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 17:41:23.275994 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:41:23.275671 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-jn5ph" Jul 27 17:41:28.275440 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:41:28.275348 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-jn5ph" podUID="72543746-44c6-417b-a368-8a35cb248743" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 17:41:33.275239 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:41:33.275199 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-jn5ph" podUID="72543746-44c6-417b-a368-8a35cb248743" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 17:41:38.274616 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:41:38.274576 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-jn5ph" podUID="72543746-44c6-417b-a368-8a35cb248743" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 17:41:41.263323 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:41:41.263295 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-jn5ph" Jul 27 17:41:41.358094 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:41:41.358059 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72543746-44c6-417b-a368-8a35cb248743-proxy-tls\") pod \"72543746-44c6-417b-a368-8a35cb248743\" (UID: \"72543746-44c6-417b-a368-8a35cb248743\") " Jul 27 17:41:41.358285 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:41:41.358119 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72543746-44c6-417b-a368-8a35cb248743-openshift-service-ca-bundle\") pod \"72543746-44c6-417b-a368-8a35cb248743\" (UID: \"72543746-44c6-417b-a368-8a35cb248743\") " Jul 27 17:41:41.358478 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:41:41.358443 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72543746-44c6-417b-a368-8a35cb248743-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "72543746-44c6-417b-a368-8a35cb248743" (UID: "72543746-44c6-417b-a368-8a35cb248743"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:41:41.360082 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:41:41.360059 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72543746-44c6-417b-a368-8a35cb248743-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "72543746-44c6-417b-a368-8a35cb248743" (UID: "72543746-44c6-417b-a368-8a35cb248743"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 17:41:41.427703 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:41:41.427622 2570 generic.go:356] "Generic (PLEG): container finished" podID="72543746-44c6-417b-a368-8a35cb248743" containerID="7fd2d1442f9e41d22c2c80ac89cb4e5e9cc3d8d88a51c16a748903fbc8278990" exitCode=0 Jul 27 17:41:41.427703 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:41:41.427666 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-jn5ph" Jul 27 17:41:41.427703 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:41:41.427677 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-jn5ph" event={"ID":"72543746-44c6-417b-a368-8a35cb248743","Type":"ContainerDied","Data":"7fd2d1442f9e41d22c2c80ac89cb4e5e9cc3d8d88a51c16a748903fbc8278990"} Jul 27 17:41:41.427916 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:41:41.427739 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-jn5ph" event={"ID":"72543746-44c6-417b-a368-8a35cb248743","Type":"ContainerDied","Data":"3baabd7aa5090ff6e3fa8f40326ae830d2dce26bc2a9e90b1737a281b988d748"} Jul 27 17:41:41.427916 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:41:41.427773 2570 scope.go:117] "RemoveContainer" containerID="7fd2d1442f9e41d22c2c80ac89cb4e5e9cc3d8d88a51c16a748903fbc8278990" Jul 27 17:41:41.435948 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:41:41.435926 2570 scope.go:117] "RemoveContainer" containerID="7fd2d1442f9e41d22c2c80ac89cb4e5e9cc3d8d88a51c16a748903fbc8278990" Jul 27 17:41:41.436226 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:41:41.436204 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fd2d1442f9e41d22c2c80ac89cb4e5e9cc3d8d88a51c16a748903fbc8278990\": container with ID starting with 7fd2d1442f9e41d22c2c80ac89cb4e5e9cc3d8d88a51c16a748903fbc8278990 not found: ID does not exist" containerID="7fd2d1442f9e41d22c2c80ac89cb4e5e9cc3d8d88a51c16a748903fbc8278990" Jul 27 17:41:41.436275 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:41:41.436240 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fd2d1442f9e41d22c2c80ac89cb4e5e9cc3d8d88a51c16a748903fbc8278990"} err="failed to get container status \"7fd2d1442f9e41d22c2c80ac89cb4e5e9cc3d8d88a51c16a748903fbc8278990\": rpc error: code = NotFound desc = could not find container \"7fd2d1442f9e41d22c2c80ac89cb4e5e9cc3d8d88a51c16a748903fbc8278990\": container with ID starting with 7fd2d1442f9e41d22c2c80ac89cb4e5e9cc3d8d88a51c16a748903fbc8278990 not found: ID does not exist" Jul 27 17:41:41.447151 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:41:41.447128 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-b84f4c8c7-jn5ph"] Jul 27 17:41:41.450790 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:41:41.450768 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-b84f4c8c7-jn5ph"] Jul 27 17:41:41.459732 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:41:41.459713 2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72543746-44c6-417b-a368-8a35cb248743-proxy-tls\") on node \"ip-10-0-140-185.ec2.internal\" DevicePath \"\"" Jul 27 17:41:41.459804 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:41:41.459735 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72543746-44c6-417b-a368-8a35cb248743-openshift-service-ca-bundle\") on node \"ip-10-0-140-185.ec2.internal\" DevicePath \"\"" Jul 27 17:41:43.129122 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:41:43.129082 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72543746-44c6-417b-a368-8a35cb248743" path="/var/lib/kubelet/pods/72543746-44c6-417b-a368-8a35cb248743/volumes" Jul 27 17:42:10.816548 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:42:10.816502 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-796c8-64c46bb88f-cghw4"] Jul 27 17:42:10.816974 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:42:10.816959 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72543746-44c6-417b-a368-8a35cb248743" containerName="model-chainer" Jul 27 17:42:10.817018 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:42:10.816976 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="72543746-44c6-417b-a368-8a35cb248743" containerName="model-chainer" Jul 27 17:42:10.817059 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:42:10.817034 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="72543746-44c6-417b-a368-8a35cb248743" containerName="model-chainer" Jul 27 17:42:10.821029 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:42:10.821007 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-796c8-64c46bb88f-cghw4" Jul 27 17:42:10.822944 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:42:10.822922 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-796c8-kube-rbac-proxy-sar-config\"" Jul 27 17:42:10.823286 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:42:10.823264 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-796c8-serving-cert\"" Jul 27 17:42:10.829576 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:42:10.829555 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-796c8-64c46bb88f-cghw4"] Jul 27 17:42:10.908977 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:42:10.908942 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ee04a4c-8891-4831-8458-bffa0c3c12e2-openshift-service-ca-bundle\") pod \"sequence-graph-796c8-64c46bb88f-cghw4\" (UID: \"0ee04a4c-8891-4831-8458-bffa0c3c12e2\") " pod="kserve-ci-e2e-test/sequence-graph-796c8-64c46bb88f-cghw4" Jul 27 17:42:10.908977 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:42:10.908980 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ee04a4c-8891-4831-8458-bffa0c3c12e2-proxy-tls\") pod \"sequence-graph-796c8-64c46bb88f-cghw4\" (UID: \"0ee04a4c-8891-4831-8458-bffa0c3c12e2\") " pod="kserve-ci-e2e-test/sequence-graph-796c8-64c46bb88f-cghw4" Jul 27 17:42:11.009646 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:42:11.009606 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ee04a4c-8891-4831-8458-bffa0c3c12e2-openshift-service-ca-bundle\") pod \"sequence-graph-796c8-64c46bb88f-cghw4\" (UID: \"0ee04a4c-8891-4831-8458-bffa0c3c12e2\") " pod="kserve-ci-e2e-test/sequence-graph-796c8-64c46bb88f-cghw4" Jul 27 17:42:11.009646 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:42:11.009649 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ee04a4c-8891-4831-8458-bffa0c3c12e2-proxy-tls\") pod \"sequence-graph-796c8-64c46bb88f-cghw4\" (UID: \"0ee04a4c-8891-4831-8458-bffa0c3c12e2\") " pod="kserve-ci-e2e-test/sequence-graph-796c8-64c46bb88f-cghw4" Jul 27 17:42:11.009873 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:42:11.009774 2570 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-796c8-serving-cert: secret "sequence-graph-796c8-serving-cert" not found Jul 27 17:42:11.009873 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:42:11.009842 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ee04a4c-8891-4831-8458-bffa0c3c12e2-proxy-tls podName:0ee04a4c-8891-4831-8458-bffa0c3c12e2 nodeName:}" failed. No retries permitted until 2026-07-27 17:42:11.509826293 +0000 UTC m=+848.951041245 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0ee04a4c-8891-4831-8458-bffa0c3c12e2-proxy-tls") pod "sequence-graph-796c8-64c46bb88f-cghw4" (UID: "0ee04a4c-8891-4831-8458-bffa0c3c12e2") : secret "sequence-graph-796c8-serving-cert" not found Jul 27 17:42:11.010225 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:42:11.010203 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ee04a4c-8891-4831-8458-bffa0c3c12e2-openshift-service-ca-bundle\") pod \"sequence-graph-796c8-64c46bb88f-cghw4\" (UID: \"0ee04a4c-8891-4831-8458-bffa0c3c12e2\") " pod="kserve-ci-e2e-test/sequence-graph-796c8-64c46bb88f-cghw4" Jul 27 17:42:11.514674 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:42:11.514639 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ee04a4c-8891-4831-8458-bffa0c3c12e2-proxy-tls\") pod \"sequence-graph-796c8-64c46bb88f-cghw4\" (UID: \"0ee04a4c-8891-4831-8458-bffa0c3c12e2\") " pod="kserve-ci-e2e-test/sequence-graph-796c8-64c46bb88f-cghw4" Jul 27 17:42:11.514848 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:42:11.514794 2570 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-796c8-serving-cert: secret "sequence-graph-796c8-serving-cert" not found Jul 27 17:42:11.514894 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:42:11.514870 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ee04a4c-8891-4831-8458-bffa0c3c12e2-proxy-tls podName:0ee04a4c-8891-4831-8458-bffa0c3c12e2 nodeName:}" failed. No retries permitted until 2026-07-27 17:42:12.514852011 +0000 UTC m=+849.956066965 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0ee04a4c-8891-4831-8458-bffa0c3c12e2-proxy-tls") pod "sequence-graph-796c8-64c46bb88f-cghw4" (UID: "0ee04a4c-8891-4831-8458-bffa0c3c12e2") : secret "sequence-graph-796c8-serving-cert" not found Jul 27 17:42:12.523089 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:42:12.523046 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ee04a4c-8891-4831-8458-bffa0c3c12e2-proxy-tls\") pod \"sequence-graph-796c8-64c46bb88f-cghw4\" (UID: \"0ee04a4c-8891-4831-8458-bffa0c3c12e2\") " pod="kserve-ci-e2e-test/sequence-graph-796c8-64c46bb88f-cghw4" Jul 27 17:42:12.525384 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:42:12.525365 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ee04a4c-8891-4831-8458-bffa0c3c12e2-proxy-tls\") pod \"sequence-graph-796c8-64c46bb88f-cghw4\" (UID: \"0ee04a4c-8891-4831-8458-bffa0c3c12e2\") " pod="kserve-ci-e2e-test/sequence-graph-796c8-64c46bb88f-cghw4" Jul 27 17:42:12.632655 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:42:12.632618 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-796c8-64c46bb88f-cghw4" Jul 27 17:42:12.760798 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:42:12.760774 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-796c8-64c46bb88f-cghw4"] Jul 27 17:42:12.762877 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:42:12.762843 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ee04a4c_8891_4831_8458_bffa0c3c12e2.slice/crio-90dbb30a303eb9891a8af75487e43a63d3d2b3afc649087f2a845de2ac7c41f5 WatchSource:0}: Error finding container 90dbb30a303eb9891a8af75487e43a63d3d2b3afc649087f2a845de2ac7c41f5: Status 404 returned error can't find the container with id 90dbb30a303eb9891a8af75487e43a63d3d2b3afc649087f2a845de2ac7c41f5 Jul 27 17:42:13.550830 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:42:13.550788 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-796c8-64c46bb88f-cghw4" event={"ID":"0ee04a4c-8891-4831-8458-bffa0c3c12e2","Type":"ContainerStarted","Data":"e49bdfce1915ef47210361a03a79934a160f300c8f60ac2a5f66178e7592c03b"} Jul 27 17:42:13.551231 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:42:13.550882 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-796c8-64c46bb88f-cghw4" Jul 27 17:42:13.551231 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:42:13.550906 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-796c8-64c46bb88f-cghw4" event={"ID":"0ee04a4c-8891-4831-8458-bffa0c3c12e2","Type":"ContainerStarted","Data":"90dbb30a303eb9891a8af75487e43a63d3d2b3afc649087f2a845de2ac7c41f5"} Jul 27 17:42:13.566111 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:42:13.566057 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-796c8-64c46bb88f-cghw4" podStartSLOduration=3.566039865 podStartE2EDuration="3.566039865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:42:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:42:13.565255295 +0000 UTC m=+851.006470281" watchObservedRunningTime="2026-07-27 17:42:13.566039865 +0000 UTC m=+851.007254867" Jul 27 17:42:19.560063 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:42:19.560033 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-796c8-64c46bb88f-cghw4" Jul 27 17:49:25.413470 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:49:25.413435 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-78126-67c9695b9f-vxqd4"] Jul 27 17:49:25.413963 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:49:25.413682 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-78126-67c9695b9f-vxqd4" podUID="96d6b210-37d9-4d67-b436-bb1d264dc3df" containerName="switch-graph-78126" containerID="cri-o://575508041b25ad26675c43253be0b823a93773e3f34664d1067dc4ada46785ff" gracePeriod=30 Jul 27 17:49:28.320490 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:49:28.320447 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-78126-67c9695b9f-vxqd4" podUID="96d6b210-37d9-4d67-b436-bb1d264dc3df" containerName="switch-graph-78126" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 17:49:33.320222 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:49:33.320184 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-78126-67c9695b9f-vxqd4" podUID="96d6b210-37d9-4d67-b436-bb1d264dc3df" containerName="switch-graph-78126" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 17:49:38.320138 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:49:38.320099 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-78126-67c9695b9f-vxqd4" podUID="96d6b210-37d9-4d67-b436-bb1d264dc3df" containerName="switch-graph-78126" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 17:49:38.320638 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:49:38.320234 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-78126-67c9695b9f-vxqd4" Jul 27 17:49:43.320411 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:49:43.320371 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-78126-67c9695b9f-vxqd4" podUID="96d6b210-37d9-4d67-b436-bb1d264dc3df" containerName="switch-graph-78126" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 17:49:48.320399 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:49:48.320356 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-78126-67c9695b9f-vxqd4" podUID="96d6b210-37d9-4d67-b436-bb1d264dc3df" containerName="switch-graph-78126" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 17:49:53.320501 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:49:53.320461 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-78126-67c9695b9f-vxqd4" podUID="96d6b210-37d9-4d67-b436-bb1d264dc3df" containerName="switch-graph-78126" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 17:49:55.559215 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:49:55.559192 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-78126-67c9695b9f-vxqd4" Jul 27 17:49:55.593005 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:49:55.592981 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/96d6b210-37d9-4d67-b436-bb1d264dc3df-proxy-tls\") pod \"96d6b210-37d9-4d67-b436-bb1d264dc3df\" (UID: \"96d6b210-37d9-4d67-b436-bb1d264dc3df\") " Jul 27 17:49:55.593155 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:49:55.593057 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96d6b210-37d9-4d67-b436-bb1d264dc3df-openshift-service-ca-bundle\") pod \"96d6b210-37d9-4d67-b436-bb1d264dc3df\" (UID: \"96d6b210-37d9-4d67-b436-bb1d264dc3df\") " Jul 27 17:49:55.593402 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:49:55.593381 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96d6b210-37d9-4d67-b436-bb1d264dc3df-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "96d6b210-37d9-4d67-b436-bb1d264dc3df" (UID: "96d6b210-37d9-4d67-b436-bb1d264dc3df"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:49:55.594994 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:49:55.594975 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96d6b210-37d9-4d67-b436-bb1d264dc3df-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "96d6b210-37d9-4d67-b436-bb1d264dc3df" (UID: "96d6b210-37d9-4d67-b436-bb1d264dc3df"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 17:49:55.694379 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:49:55.694295 2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/96d6b210-37d9-4d67-b436-bb1d264dc3df-proxy-tls\") on node \"ip-10-0-140-185.ec2.internal\" DevicePath \"\"" Jul 27 17:49:55.694379 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:49:55.694321 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96d6b210-37d9-4d67-b436-bb1d264dc3df-openshift-service-ca-bundle\") on node \"ip-10-0-140-185.ec2.internal\" DevicePath \"\"" Jul 27 17:49:56.353570 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:49:56.353533 2570 generic.go:356] "Generic (PLEG): container finished" podID="96d6b210-37d9-4d67-b436-bb1d264dc3df" containerID="575508041b25ad26675c43253be0b823a93773e3f34664d1067dc4ada46785ff" exitCode=0 Jul 27 17:49:56.353785 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:49:56.353589 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-78126-67c9695b9f-vxqd4" Jul 27 17:49:56.353785 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:49:56.353631 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-78126-67c9695b9f-vxqd4" event={"ID":"96d6b210-37d9-4d67-b436-bb1d264dc3df","Type":"ContainerDied","Data":"575508041b25ad26675c43253be0b823a93773e3f34664d1067dc4ada46785ff"} Jul 27 17:49:56.353785 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:49:56.353693 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-78126-67c9695b9f-vxqd4" event={"ID":"96d6b210-37d9-4d67-b436-bb1d264dc3df","Type":"ContainerDied","Data":"f82db4981c0fbb3bee8371f44a619dbc4e505dd280b5394f763e45c5050fbdd3"} Jul 27 17:49:56.353785 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:49:56.353716 2570 scope.go:117] "RemoveContainer" containerID="575508041b25ad26675c43253be0b823a93773e3f34664d1067dc4ada46785ff" Jul 27 17:49:56.362652 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:49:56.362634 2570 scope.go:117] "RemoveContainer" containerID="575508041b25ad26675c43253be0b823a93773e3f34664d1067dc4ada46785ff" Jul 27 17:49:56.362877 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:49:56.362860 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"575508041b25ad26675c43253be0b823a93773e3f34664d1067dc4ada46785ff\": container with ID starting with 575508041b25ad26675c43253be0b823a93773e3f34664d1067dc4ada46785ff not found: ID does not exist" containerID="575508041b25ad26675c43253be0b823a93773e3f34664d1067dc4ada46785ff" Jul 27 17:49:56.362922 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:49:56.362886 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"575508041b25ad26675c43253be0b823a93773e3f34664d1067dc4ada46785ff"} err="failed to get container status \"575508041b25ad26675c43253be0b823a93773e3f34664d1067dc4ada46785ff\": rpc error: code = NotFound desc = could not find container \"575508041b25ad26675c43253be0b823a93773e3f34664d1067dc4ada46785ff\": container with ID starting with 575508041b25ad26675c43253be0b823a93773e3f34664d1067dc4ada46785ff not found: ID does not exist" Jul 27 17:49:56.379095 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:49:56.377709 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-78126-67c9695b9f-vxqd4"] Jul 27 17:49:56.382457 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:49:56.382431 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-78126-67c9695b9f-vxqd4"] Jul 27 17:49:57.128929 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:49:57.128895 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96d6b210-37d9-4d67-b436-bb1d264dc3df" path="/var/lib/kubelet/pods/96d6b210-37d9-4d67-b436-bb1d264dc3df/volumes" Jul 27 17:50:25.504416 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:50:25.504383 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-796c8-64c46bb88f-cghw4"] Jul 27 17:50:25.504945 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:50:25.504593 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-796c8-64c46bb88f-cghw4" podUID="0ee04a4c-8891-4831-8458-bffa0c3c12e2" containerName="sequence-graph-796c8" containerID="cri-o://e49bdfce1915ef47210361a03a79934a160f300c8f60ac2a5f66178e7592c03b" gracePeriod=30 Jul 27 17:50:25.666180 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:50:25.666151 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-37f6e-5ffd8db85f-4hc4b"] Jul 27 17:50:25.666484 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:50:25.666460 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96d6b210-37d9-4d67-b436-bb1d264dc3df" containerName="switch-graph-78126" Jul 27 17:50:25.666484 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:50:25.666476 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="96d6b210-37d9-4d67-b436-bb1d264dc3df" containerName="switch-graph-78126" Jul 27 17:50:25.666701 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:50:25.666591 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="96d6b210-37d9-4d67-b436-bb1d264dc3df" containerName="switch-graph-78126" Jul 27 17:50:25.670603 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:50:25.670584 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-37f6e-5ffd8db85f-4hc4b" Jul 27 17:50:25.674675 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:50:25.674655 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-37f6e-kube-rbac-proxy-sar-config\"" Jul 27 17:50:25.674950 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:50:25.674935 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-37f6e-serving-cert\"" Jul 27 17:50:25.685500 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:50:25.685477 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-37f6e-5ffd8db85f-4hc4b"] Jul 27 17:50:25.844130 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:50:25.844039 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08f10a0e-c005-4ce8-b5af-be5137cfb3dc-openshift-service-ca-bundle\") pod \"ensemble-graph-37f6e-5ffd8db85f-4hc4b\" (UID: \"08f10a0e-c005-4ce8-b5af-be5137cfb3dc\") " pod="kserve-ci-e2e-test/ensemble-graph-37f6e-5ffd8db85f-4hc4b" Jul 27 17:50:25.844130 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:50:25.844087 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08f10a0e-c005-4ce8-b5af-be5137cfb3dc-proxy-tls\") pod \"ensemble-graph-37f6e-5ffd8db85f-4hc4b\" (UID: \"08f10a0e-c005-4ce8-b5af-be5137cfb3dc\") " pod="kserve-ci-e2e-test/ensemble-graph-37f6e-5ffd8db85f-4hc4b" Jul 27 17:50:25.945223 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:50:25.945189 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08f10a0e-c005-4ce8-b5af-be5137cfb3dc-openshift-service-ca-bundle\") pod \"ensemble-graph-37f6e-5ffd8db85f-4hc4b\" (UID: \"08f10a0e-c005-4ce8-b5af-be5137cfb3dc\") " pod="kserve-ci-e2e-test/ensemble-graph-37f6e-5ffd8db85f-4hc4b" Jul 27 17:50:25.945223 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:50:25.945233 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08f10a0e-c005-4ce8-b5af-be5137cfb3dc-proxy-tls\") pod \"ensemble-graph-37f6e-5ffd8db85f-4hc4b\" (UID: \"08f10a0e-c005-4ce8-b5af-be5137cfb3dc\") " pod="kserve-ci-e2e-test/ensemble-graph-37f6e-5ffd8db85f-4hc4b" Jul 27 17:50:25.945850 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:50:25.945829 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08f10a0e-c005-4ce8-b5af-be5137cfb3dc-openshift-service-ca-bundle\") pod \"ensemble-graph-37f6e-5ffd8db85f-4hc4b\" (UID: \"08f10a0e-c005-4ce8-b5af-be5137cfb3dc\") " pod="kserve-ci-e2e-test/ensemble-graph-37f6e-5ffd8db85f-4hc4b" Jul 27 17:50:25.947799 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:50:25.947778 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08f10a0e-c005-4ce8-b5af-be5137cfb3dc-proxy-tls\") pod \"ensemble-graph-37f6e-5ffd8db85f-4hc4b\" (UID: \"08f10a0e-c005-4ce8-b5af-be5137cfb3dc\") " pod="kserve-ci-e2e-test/ensemble-graph-37f6e-5ffd8db85f-4hc4b" Jul 27 17:50:25.984571 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:50:25.984547 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-37f6e-5ffd8db85f-4hc4b" Jul 27 17:50:26.111600 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:50:26.111578 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-37f6e-5ffd8db85f-4hc4b"] Jul 27 17:50:26.113971 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:50:26.113941 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08f10a0e_c005_4ce8_b5af_be5137cfb3dc.slice/crio-c2038b39229ed7f484f8cda49165650a96e92df20ec0ac588de1843092ecf915 WatchSource:0}: Error finding container c2038b39229ed7f484f8cda49165650a96e92df20ec0ac588de1843092ecf915: Status 404 returned error can't find the container with id c2038b39229ed7f484f8cda49165650a96e92df20ec0ac588de1843092ecf915 Jul 27 17:50:26.472793 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:50:26.472702 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-37f6e-5ffd8db85f-4hc4b" event={"ID":"08f10a0e-c005-4ce8-b5af-be5137cfb3dc","Type":"ContainerStarted","Data":"40d56c7f646996a41e11b001f8e9bb247a21e70ccfec81b70b19592e799140f2"} Jul 27 17:50:26.472793 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:50:26.472764 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-37f6e-5ffd8db85f-4hc4b" event={"ID":"08f10a0e-c005-4ce8-b5af-be5137cfb3dc","Type":"ContainerStarted","Data":"c2038b39229ed7f484f8cda49165650a96e92df20ec0ac588de1843092ecf915"} Jul 27 17:50:26.472977 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:50:26.472845 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-37f6e-5ffd8db85f-4hc4b" Jul 27 17:50:26.488430 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:50:26.488387 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-37f6e-5ffd8db85f-4hc4b" podStartSLOduration=1.488371783 podStartE2EDuration="1.488371783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:50:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:50:26.48718792 +0000 UTC m=+1343.928402894" watchObservedRunningTime="2026-07-27 17:50:26.488371783 +0000 UTC m=+1343.929586757" Jul 27 17:50:29.559164 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:50:29.559129 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-796c8-64c46bb88f-cghw4" podUID="0ee04a4c-8891-4831-8458-bffa0c3c12e2" containerName="sequence-graph-796c8" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 17:50:32.482629 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:50:32.482601 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-37f6e-5ffd8db85f-4hc4b" Jul 27 17:50:34.558970 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:50:34.558927 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-796c8-64c46bb88f-cghw4" podUID="0ee04a4c-8891-4831-8458-bffa0c3c12e2" containerName="sequence-graph-796c8" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 17:50:35.709476 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:50:35.709445 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-37f6e-5ffd8db85f-4hc4b"] Jul 27 17:50:35.709863 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:50:35.709697 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-37f6e-5ffd8db85f-4hc4b" podUID="08f10a0e-c005-4ce8-b5af-be5137cfb3dc" containerName="ensemble-graph-37f6e" containerID="cri-o://40d56c7f646996a41e11b001f8e9bb247a21e70ccfec81b70b19592e799140f2" gracePeriod=30 Jul 27 17:50:37.481296 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:50:37.481253 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-37f6e-5ffd8db85f-4hc4b" podUID="08f10a0e-c005-4ce8-b5af-be5137cfb3dc" containerName="ensemble-graph-37f6e" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 17:50:39.559212 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:50:39.559168 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-796c8-64c46bb88f-cghw4" podUID="0ee04a4c-8891-4831-8458-bffa0c3c12e2" containerName="sequence-graph-796c8" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 17:50:39.559657 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:50:39.559289 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-796c8-64c46bb88f-cghw4" Jul 27 17:50:42.480335 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:50:42.480299 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-37f6e-5ffd8db85f-4hc4b" podUID="08f10a0e-c005-4ce8-b5af-be5137cfb3dc" containerName="ensemble-graph-37f6e" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 17:50:44.558996 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:50:44.558955 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-796c8-64c46bb88f-cghw4" podUID="0ee04a4c-8891-4831-8458-bffa0c3c12e2" containerName="sequence-graph-796c8" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 17:50:47.480538 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:50:47.480492 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-37f6e-5ffd8db85f-4hc4b" podUID="08f10a0e-c005-4ce8-b5af-be5137cfb3dc" containerName="ensemble-graph-37f6e" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 17:50:47.480982 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:50:47.480622 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-37f6e-5ffd8db85f-4hc4b" Jul 27 17:50:49.559306 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:50:49.559268 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-796c8-64c46bb88f-cghw4" podUID="0ee04a4c-8891-4831-8458-bffa0c3c12e2" containerName="sequence-graph-796c8" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 17:50:52.480756 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:50:52.480723 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-37f6e-5ffd8db85f-4hc4b" podUID="08f10a0e-c005-4ce8-b5af-be5137cfb3dc" containerName="ensemble-graph-37f6e" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 17:50:54.558883 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:50:54.558846 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-796c8-64c46bb88f-cghw4" podUID="0ee04a4c-8891-4831-8458-bffa0c3c12e2" containerName="sequence-graph-796c8" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 17:50:55.587823 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:50:55.587792 2570 generic.go:356] "Generic (PLEG): container finished" podID="0ee04a4c-8891-4831-8458-bffa0c3c12e2" containerID="e49bdfce1915ef47210361a03a79934a160f300c8f60ac2a5f66178e7592c03b" exitCode=0 Jul 27 17:50:55.588258 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:50:55.587833 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-796c8-64c46bb88f-cghw4" event={"ID":"0ee04a4c-8891-4831-8458-bffa0c3c12e2","Type":"ContainerDied","Data":"e49bdfce1915ef47210361a03a79934a160f300c8f60ac2a5f66178e7592c03b"} Jul 27 17:50:55.647631 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:50:55.647609 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-796c8-64c46bb88f-cghw4" Jul 27 17:50:55.696185 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:50:55.696157 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ee04a4c-8891-4831-8458-bffa0c3c12e2-openshift-service-ca-bundle\") pod \"0ee04a4c-8891-4831-8458-bffa0c3c12e2\" (UID: \"0ee04a4c-8891-4831-8458-bffa0c3c12e2\") " Jul 27 17:50:55.696185 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:50:55.696188 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ee04a4c-8891-4831-8458-bffa0c3c12e2-proxy-tls\") pod \"0ee04a4c-8891-4831-8458-bffa0c3c12e2\" (UID: \"0ee04a4c-8891-4831-8458-bffa0c3c12e2\") " Jul 27 17:50:55.696548 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:50:55.696503 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ee04a4c-8891-4831-8458-bffa0c3c12e2-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "0ee04a4c-8891-4831-8458-bffa0c3c12e2" (UID: "0ee04a4c-8891-4831-8458-bffa0c3c12e2"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:50:55.698196 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:50:55.698165 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ee04a4c-8891-4831-8458-bffa0c3c12e2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0ee04a4c-8891-4831-8458-bffa0c3c12e2" (UID: "0ee04a4c-8891-4831-8458-bffa0c3c12e2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 17:50:55.797433 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:50:55.797352 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ee04a4c-8891-4831-8458-bffa0c3c12e2-openshift-service-ca-bundle\") on node \"ip-10-0-140-185.ec2.internal\" DevicePath \"\"" Jul 27 17:50:55.797433 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:50:55.797381 2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ee04a4c-8891-4831-8458-bffa0c3c12e2-proxy-tls\") on node \"ip-10-0-140-185.ec2.internal\" DevicePath \"\"" Jul 27 17:50:56.593089 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:50:56.593059 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-796c8-64c46bb88f-cghw4" Jul 27 17:50:56.593571 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:50:56.593055 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-796c8-64c46bb88f-cghw4" event={"ID":"0ee04a4c-8891-4831-8458-bffa0c3c12e2","Type":"ContainerDied","Data":"90dbb30a303eb9891a8af75487e43a63d3d2b3afc649087f2a845de2ac7c41f5"} Jul 27 17:50:56.593571 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:50:56.593199 2570 scope.go:117] "RemoveContainer" containerID="e49bdfce1915ef47210361a03a79934a160f300c8f60ac2a5f66178e7592c03b" Jul 27 17:50:56.613266 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:50:56.613240 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-796c8-64c46bb88f-cghw4"] Jul 27 17:50:56.618800 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:50:56.618780 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-796c8-64c46bb88f-cghw4"] Jul 27 17:50:57.128398 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:50:57.128364 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ee04a4c-8891-4831-8458-bffa0c3c12e2" path="/var/lib/kubelet/pods/0ee04a4c-8891-4831-8458-bffa0c3c12e2/volumes" Jul 27 17:50:57.481133 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:50:57.481050 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-37f6e-5ffd8db85f-4hc4b" podUID="08f10a0e-c005-4ce8-b5af-be5137cfb3dc" containerName="ensemble-graph-37f6e" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 17:51:02.480964 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:02.480928 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-37f6e-5ffd8db85f-4hc4b" podUID="08f10a0e-c005-4ce8-b5af-be5137cfb3dc" containerName="ensemble-graph-37f6e" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 17:51:05.855562 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:05.855511 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-37f6e-5ffd8db85f-4hc4b" Jul 27 17:51:05.978871 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:05.978780 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08f10a0e-c005-4ce8-b5af-be5137cfb3dc-openshift-service-ca-bundle\") pod \"08f10a0e-c005-4ce8-b5af-be5137cfb3dc\" (UID: \"08f10a0e-c005-4ce8-b5af-be5137cfb3dc\") " Jul 27 17:51:05.979097 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:05.978890 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08f10a0e-c005-4ce8-b5af-be5137cfb3dc-proxy-tls\") pod \"08f10a0e-c005-4ce8-b5af-be5137cfb3dc\" (UID: \"08f10a0e-c005-4ce8-b5af-be5137cfb3dc\") " Jul 27 17:51:05.979164 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:05.979117 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08f10a0e-c005-4ce8-b5af-be5137cfb3dc-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "08f10a0e-c005-4ce8-b5af-be5137cfb3dc" (UID: "08f10a0e-c005-4ce8-b5af-be5137cfb3dc"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:51:05.980801 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:05.980778 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08f10a0e-c005-4ce8-b5af-be5137cfb3dc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "08f10a0e-c005-4ce8-b5af-be5137cfb3dc" (UID: "08f10a0e-c005-4ce8-b5af-be5137cfb3dc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 17:51:06.080044 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:06.079998 2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08f10a0e-c005-4ce8-b5af-be5137cfb3dc-proxy-tls\") on node \"ip-10-0-140-185.ec2.internal\" DevicePath \"\"" Jul 27 17:51:06.080044 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:06.080040 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08f10a0e-c005-4ce8-b5af-be5137cfb3dc-openshift-service-ca-bundle\") on node \"ip-10-0-140-185.ec2.internal\" DevicePath \"\"" Jul 27 17:51:06.630286 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:06.630252 2570 generic.go:356] "Generic (PLEG): container finished" podID="08f10a0e-c005-4ce8-b5af-be5137cfb3dc" containerID="40d56c7f646996a41e11b001f8e9bb247a21e70ccfec81b70b19592e799140f2" exitCode=0 Jul 27 17:51:06.630286 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:06.630288 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-37f6e-5ffd8db85f-4hc4b" event={"ID":"08f10a0e-c005-4ce8-b5af-be5137cfb3dc","Type":"ContainerDied","Data":"40d56c7f646996a41e11b001f8e9bb247a21e70ccfec81b70b19592e799140f2"} Jul 27 17:51:06.630514 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:06.630297 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-37f6e-5ffd8db85f-4hc4b" Jul 27 17:51:06.630514 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:06.630325 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-37f6e-5ffd8db85f-4hc4b" event={"ID":"08f10a0e-c005-4ce8-b5af-be5137cfb3dc","Type":"ContainerDied","Data":"c2038b39229ed7f484f8cda49165650a96e92df20ec0ac588de1843092ecf915"} Jul 27 17:51:06.630514 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:06.630347 2570 scope.go:117] "RemoveContainer" containerID="40d56c7f646996a41e11b001f8e9bb247a21e70ccfec81b70b19592e799140f2" Jul 27 17:51:06.641719 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:06.639479 2570 scope.go:117] "RemoveContainer" containerID="40d56c7f646996a41e11b001f8e9bb247a21e70ccfec81b70b19592e799140f2" Jul 27 17:51:06.641719 ip-10-0-140-185 kubenswrapper[2570]: E0727 17:51:06.640002 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40d56c7f646996a41e11b001f8e9bb247a21e70ccfec81b70b19592e799140f2\": container with ID starting with 40d56c7f646996a41e11b001f8e9bb247a21e70ccfec81b70b19592e799140f2 not found: ID does not exist" containerID="40d56c7f646996a41e11b001f8e9bb247a21e70ccfec81b70b19592e799140f2" Jul 27 17:51:06.641719 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:06.640032 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40d56c7f646996a41e11b001f8e9bb247a21e70ccfec81b70b19592e799140f2"} err="failed to get container status \"40d56c7f646996a41e11b001f8e9bb247a21e70ccfec81b70b19592e799140f2\": rpc error: code = NotFound desc = could not find container \"40d56c7f646996a41e11b001f8e9bb247a21e70ccfec81b70b19592e799140f2\": container with ID starting with 40d56c7f646996a41e11b001f8e9bb247a21e70ccfec81b70b19592e799140f2 not found: ID does not exist" Jul 27 17:51:06.651627 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:06.651600 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-37f6e-5ffd8db85f-4hc4b"] Jul 27 17:51:06.655943 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:06.655920 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-37f6e-5ffd8db85f-4hc4b"] Jul 27 17:51:07.128999 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:07.128969 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08f10a0e-c005-4ce8-b5af-be5137cfb3dc" path="/var/lib/kubelet/pods/08f10a0e-c005-4ce8-b5af-be5137cfb3dc/volumes" Jul 27 17:51:25.707738 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:25.707708 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-60da7-6cdf8f456b-c6lzx"] Jul 27 17:51:25.708264 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:25.708168 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ee04a4c-8891-4831-8458-bffa0c3c12e2" containerName="sequence-graph-796c8" Jul 27 17:51:25.708264 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:25.708186 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ee04a4c-8891-4831-8458-bffa0c3c12e2" containerName="sequence-graph-796c8" Jul 27 17:51:25.708264 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:25.708209 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08f10a0e-c005-4ce8-b5af-be5137cfb3dc" containerName="ensemble-graph-37f6e" Jul 27 17:51:25.708264 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:25.708217 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="08f10a0e-c005-4ce8-b5af-be5137cfb3dc" containerName="ensemble-graph-37f6e" Jul 27 17:51:25.708468 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:25.708290 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="08f10a0e-c005-4ce8-b5af-be5137cfb3dc" containerName="ensemble-graph-37f6e" Jul 27 17:51:25.708468 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:25.708307 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ee04a4c-8891-4831-8458-bffa0c3c12e2" containerName="sequence-graph-796c8" Jul 27 17:51:25.712504 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:25.712485 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-60da7-6cdf8f456b-c6lzx" Jul 27 17:51:25.714559 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:25.714539 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-60da7-kube-rbac-proxy-sar-config\"" Jul 27 17:51:25.714660 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:25.714558 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-60da7-serving-cert\"" Jul 27 17:51:25.714721 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:25.714658 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-x6c7x\"" Jul 27 17:51:25.714893 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:25.714876 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 27 17:51:25.718858 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:25.718836 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-60da7-6cdf8f456b-c6lzx"] Jul 27 17:51:25.848333 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:25.848295 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99be597e-3bb2-4211-94be-925afcf26e48-openshift-service-ca-bundle\") pod \"sequence-graph-60da7-6cdf8f456b-c6lzx\" (UID: \"99be597e-3bb2-4211-94be-925afcf26e48\") " pod="kserve-ci-e2e-test/sequence-graph-60da7-6cdf8f456b-c6lzx" Jul 27 17:51:25.848558 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:25.848348 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99be597e-3bb2-4211-94be-925afcf26e48-proxy-tls\") pod \"sequence-graph-60da7-6cdf8f456b-c6lzx\" (UID: \"99be597e-3bb2-4211-94be-925afcf26e48\") " pod="kserve-ci-e2e-test/sequence-graph-60da7-6cdf8f456b-c6lzx" Jul 27 17:51:25.948885 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:25.948850 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99be597e-3bb2-4211-94be-925afcf26e48-openshift-service-ca-bundle\") pod \"sequence-graph-60da7-6cdf8f456b-c6lzx\" (UID: \"99be597e-3bb2-4211-94be-925afcf26e48\") " pod="kserve-ci-e2e-test/sequence-graph-60da7-6cdf8f456b-c6lzx" Jul 27 17:51:25.949095 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:25.948913 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99be597e-3bb2-4211-94be-925afcf26e48-proxy-tls\") pod \"sequence-graph-60da7-6cdf8f456b-c6lzx\" (UID: \"99be597e-3bb2-4211-94be-925afcf26e48\") " pod="kserve-ci-e2e-test/sequence-graph-60da7-6cdf8f456b-c6lzx" Jul 27 17:51:25.949787 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:25.949764 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99be597e-3bb2-4211-94be-925afcf26e48-openshift-service-ca-bundle\") pod \"sequence-graph-60da7-6cdf8f456b-c6lzx\" (UID: \"99be597e-3bb2-4211-94be-925afcf26e48\") " pod="kserve-ci-e2e-test/sequence-graph-60da7-6cdf8f456b-c6lzx" Jul 27 17:51:25.951265 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:25.951247 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99be597e-3bb2-4211-94be-925afcf26e48-proxy-tls\") pod \"sequence-graph-60da7-6cdf8f456b-c6lzx\" (UID: \"99be597e-3bb2-4211-94be-925afcf26e48\") " pod="kserve-ci-e2e-test/sequence-graph-60da7-6cdf8f456b-c6lzx" Jul 27 17:51:26.023274 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:26.023248 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-60da7-6cdf8f456b-c6lzx" Jul 27 17:51:26.146234 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:26.146213 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-60da7-6cdf8f456b-c6lzx"] Jul 27 17:51:26.148474 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:51:26.148448 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99be597e_3bb2_4211_94be_925afcf26e48.slice/crio-33c008e43790c3c2fa33f30e86eb3304ad4b3619fdee18852f74786205174282 WatchSource:0}: Error finding container 33c008e43790c3c2fa33f30e86eb3304ad4b3619fdee18852f74786205174282: Status 404 returned error can't find the container with id 33c008e43790c3c2fa33f30e86eb3304ad4b3619fdee18852f74786205174282 Jul 27 17:51:26.709910 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:26.709880 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-60da7-6cdf8f456b-c6lzx" event={"ID":"99be597e-3bb2-4211-94be-925afcf26e48","Type":"ContainerStarted","Data":"3e0f6c9a4e65772692680cea2ecfafc06e47c29cfb8cbc6d97700d70a10ecf9c"} Jul 27 17:51:26.710347 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:26.709931 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-60da7-6cdf8f456b-c6lzx" event={"ID":"99be597e-3bb2-4211-94be-925afcf26e48","Type":"ContainerStarted","Data":"33c008e43790c3c2fa33f30e86eb3304ad4b3619fdee18852f74786205174282"} Jul 27 17:51:26.710347 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:26.710005 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-60da7-6cdf8f456b-c6lzx" Jul 27 17:51:26.727375 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:26.727332 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-60da7-6cdf8f456b-c6lzx" podStartSLOduration=1.7273170169999998 podStartE2EDuration="1.727317017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:51:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:51:26.726454543 +0000 UTC m=+1404.167669517" watchObservedRunningTime="2026-07-27 17:51:26.727317017 +0000 UTC m=+1404.168531987" Jul 27 17:51:32.720020 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:32.719986 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-60da7-6cdf8f456b-c6lzx" Jul 27 17:51:35.778006 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:35.777973 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-60da7-6cdf8f456b-c6lzx"] Jul 27 17:51:35.778375 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:35.778181 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-60da7-6cdf8f456b-c6lzx" podUID="99be597e-3bb2-4211-94be-925afcf26e48" containerName="sequence-graph-60da7" containerID="cri-o://3e0f6c9a4e65772692680cea2ecfafc06e47c29cfb8cbc6d97700d70a10ecf9c" gracePeriod=30 Jul 27 17:51:35.940400 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:35.940368 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-28de7-866ccb47df-82qn8"] Jul 27 17:51:35.944096 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:35.944072 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-28de7-866ccb47df-82qn8" Jul 27 17:51:35.946034 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:35.946011 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-28de7-serving-cert\"" Jul 27 17:51:35.946264 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:35.946245 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-28de7-kube-rbac-proxy-sar-config\"" Jul 27 17:51:35.949558 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:35.949536 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-28de7-866ccb47df-82qn8"] Jul 27 17:51:36.027577 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:36.027539 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28bcd208-9c5c-4e7b-89e8-efe4a0edd09f-proxy-tls\") pod \"ensemble-graph-28de7-866ccb47df-82qn8\" (UID: \"28bcd208-9c5c-4e7b-89e8-efe4a0edd09f\") " pod="kserve-ci-e2e-test/ensemble-graph-28de7-866ccb47df-82qn8" Jul 27 17:51:36.027756 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:36.027600 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28bcd208-9c5c-4e7b-89e8-efe4a0edd09f-openshift-service-ca-bundle\") pod \"ensemble-graph-28de7-866ccb47df-82qn8\" (UID: \"28bcd208-9c5c-4e7b-89e8-efe4a0edd09f\") " pod="kserve-ci-e2e-test/ensemble-graph-28de7-866ccb47df-82qn8" Jul 27 17:51:36.128678 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:36.128587 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28bcd208-9c5c-4e7b-89e8-efe4a0edd09f-proxy-tls\") pod \"ensemble-graph-28de7-866ccb47df-82qn8\" (UID: \"28bcd208-9c5c-4e7b-89e8-efe4a0edd09f\") " pod="kserve-ci-e2e-test/ensemble-graph-28de7-866ccb47df-82qn8" Jul 27 17:51:36.128678 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:36.128634 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28bcd208-9c5c-4e7b-89e8-efe4a0edd09f-openshift-service-ca-bundle\") pod \"ensemble-graph-28de7-866ccb47df-82qn8\" (UID: \"28bcd208-9c5c-4e7b-89e8-efe4a0edd09f\") " pod="kserve-ci-e2e-test/ensemble-graph-28de7-866ccb47df-82qn8" Jul 27 17:51:36.129224 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:36.129205 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28bcd208-9c5c-4e7b-89e8-efe4a0edd09f-openshift-service-ca-bundle\") pod \"ensemble-graph-28de7-866ccb47df-82qn8\" (UID: \"28bcd208-9c5c-4e7b-89e8-efe4a0edd09f\") " pod="kserve-ci-e2e-test/ensemble-graph-28de7-866ccb47df-82qn8" Jul 27 17:51:36.131018 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:36.131000 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28bcd208-9c5c-4e7b-89e8-efe4a0edd09f-proxy-tls\") pod \"ensemble-graph-28de7-866ccb47df-82qn8\" (UID: \"28bcd208-9c5c-4e7b-89e8-efe4a0edd09f\") " pod="kserve-ci-e2e-test/ensemble-graph-28de7-866ccb47df-82qn8" Jul 27 17:51:36.255444 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:36.255404 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-28de7-866ccb47df-82qn8" Jul 27 17:51:36.381465 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:36.381445 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-28de7-866ccb47df-82qn8"] Jul 27 17:51:36.383738 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:51:36.383712 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28bcd208_9c5c_4e7b_89e8_efe4a0edd09f.slice/crio-884620ddaebd28434b7b7a683a6314ab30a70f1c61b6c22cd3477154b7431c9f WatchSource:0}: Error finding container 884620ddaebd28434b7b7a683a6314ab30a70f1c61b6c22cd3477154b7431c9f: Status 404 returned error can't find the container with id 884620ddaebd28434b7b7a683a6314ab30a70f1c61b6c22cd3477154b7431c9f Jul 27 17:51:36.749696 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:36.749601 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-28de7-866ccb47df-82qn8" event={"ID":"28bcd208-9c5c-4e7b-89e8-efe4a0edd09f","Type":"ContainerStarted","Data":"0985f4701160a52cbc03eec61f2879ee30e916e8d6fb6b2a3e6613f6a046a14e"} Jul 27 17:51:36.749696 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:36.749662 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-28de7-866ccb47df-82qn8" event={"ID":"28bcd208-9c5c-4e7b-89e8-efe4a0edd09f","Type":"ContainerStarted","Data":"884620ddaebd28434b7b7a683a6314ab30a70f1c61b6c22cd3477154b7431c9f"} Jul 27 17:51:36.749696 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:36.749683 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-28de7-866ccb47df-82qn8" Jul 27 17:51:36.766798 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:36.766749 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-28de7-866ccb47df-82qn8" podStartSLOduration=1.766732043 podStartE2EDuration="1.766732043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:51:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:51:36.766028105 +0000 UTC m=+1414.207243080" watchObservedRunningTime="2026-07-27 17:51:36.766732043 +0000 UTC m=+1414.207947017" Jul 27 17:51:37.718575 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:37.718534 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-60da7-6cdf8f456b-c6lzx" podUID="99be597e-3bb2-4211-94be-925afcf26e48" containerName="sequence-graph-60da7" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 17:51:42.718479 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:42.718440 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-60da7-6cdf8f456b-c6lzx" podUID="99be597e-3bb2-4211-94be-925afcf26e48" containerName="sequence-graph-60da7" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 17:51:42.759748 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:42.759722 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-28de7-866ccb47df-82qn8" Jul 27 17:51:47.722707 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:47.722657 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-60da7-6cdf8f456b-c6lzx" podUID="99be597e-3bb2-4211-94be-925afcf26e48" containerName="sequence-graph-60da7" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 17:51:47.723112 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:47.722818 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-60da7-6cdf8f456b-c6lzx" Jul 27 17:51:52.718080 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:52.718041 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-60da7-6cdf8f456b-c6lzx" podUID="99be597e-3bb2-4211-94be-925afcf26e48" containerName="sequence-graph-60da7" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 17:51:57.718460 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:51:57.718373 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-60da7-6cdf8f456b-c6lzx" podUID="99be597e-3bb2-4211-94be-925afcf26e48" containerName="sequence-graph-60da7" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 17:52:02.718447 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:52:02.718407 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-60da7-6cdf8f456b-c6lzx" podUID="99be597e-3bb2-4211-94be-925afcf26e48" containerName="sequence-graph-60da7" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 17:52:05.866421 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:52:05.866387 2570 generic.go:356] "Generic (PLEG): container finished" podID="99be597e-3bb2-4211-94be-925afcf26e48" containerID="3e0f6c9a4e65772692680cea2ecfafc06e47c29cfb8cbc6d97700d70a10ecf9c" exitCode=137 Jul 27 17:52:05.866864 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:52:05.866458 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-60da7-6cdf8f456b-c6lzx" event={"ID":"99be597e-3bb2-4211-94be-925afcf26e48","Type":"ContainerDied","Data":"3e0f6c9a4e65772692680cea2ecfafc06e47c29cfb8cbc6d97700d70a10ecf9c"} Jul 27 17:52:05.960309 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:52:05.960287 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-60da7-6cdf8f456b-c6lzx" Jul 27 17:52:06.089613 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:52:06.089504 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99be597e-3bb2-4211-94be-925afcf26e48-proxy-tls\") pod \"99be597e-3bb2-4211-94be-925afcf26e48\" (UID: \"99be597e-3bb2-4211-94be-925afcf26e48\") " Jul 27 17:52:06.089613 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:52:06.089571 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99be597e-3bb2-4211-94be-925afcf26e48-openshift-service-ca-bundle\") pod \"99be597e-3bb2-4211-94be-925afcf26e48\" (UID: \"99be597e-3bb2-4211-94be-925afcf26e48\") " Jul 27 17:52:06.089938 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:52:06.089913 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99be597e-3bb2-4211-94be-925afcf26e48-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "99be597e-3bb2-4211-94be-925afcf26e48" (UID: "99be597e-3bb2-4211-94be-925afcf26e48"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 17:52:06.091554 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:52:06.091513 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99be597e-3bb2-4211-94be-925afcf26e48-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "99be597e-3bb2-4211-94be-925afcf26e48" (UID: "99be597e-3bb2-4211-94be-925afcf26e48"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 17:52:06.190458 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:52:06.190425 2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99be597e-3bb2-4211-94be-925afcf26e48-proxy-tls\") on node \"ip-10-0-140-185.ec2.internal\" DevicePath \"\"" Jul 27 17:52:06.190458 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:52:06.190455 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99be597e-3bb2-4211-94be-925afcf26e48-openshift-service-ca-bundle\") on node \"ip-10-0-140-185.ec2.internal\" DevicePath \"\"" Jul 27 17:52:06.871041 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:52:06.871013 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-60da7-6cdf8f456b-c6lzx" Jul 27 17:52:06.871041 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:52:06.871022 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-60da7-6cdf8f456b-c6lzx" event={"ID":"99be597e-3bb2-4211-94be-925afcf26e48","Type":"ContainerDied","Data":"33c008e43790c3c2fa33f30e86eb3304ad4b3619fdee18852f74786205174282"} Jul 27 17:52:06.871493 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:52:06.871077 2570 scope.go:117] "RemoveContainer" containerID="3e0f6c9a4e65772692680cea2ecfafc06e47c29cfb8cbc6d97700d70a10ecf9c" Jul 27 17:52:06.893237 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:52:06.893213 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-60da7-6cdf8f456b-c6lzx"] Jul 27 17:52:06.896599 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:52:06.896577 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-60da7-6cdf8f456b-c6lzx"] Jul 27 17:52:07.128471 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:52:07.128390 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99be597e-3bb2-4211-94be-925afcf26e48" path="/var/lib/kubelet/pods/99be597e-3bb2-4211-94be-925afcf26e48/volumes" Jul 27 17:52:36.036823 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:52:36.036794 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-19a85-bc74c9775-9pdst"] Jul 27 17:52:36.037287 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:52:36.037113 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99be597e-3bb2-4211-94be-925afcf26e48" containerName="sequence-graph-60da7" Jul 27 17:52:36.037287 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:52:36.037124 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="99be597e-3bb2-4211-94be-925afcf26e48" containerName="sequence-graph-60da7" Jul 27 17:52:36.037287 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:52:36.037184 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="99be597e-3bb2-4211-94be-925afcf26e48" containerName="sequence-graph-60da7" Jul 27 17:52:36.039999 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:52:36.039979 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-19a85-bc74c9775-9pdst" Jul 27 17:52:36.041964 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:52:36.041947 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-19a85-serving-cert\"" Jul 27 17:52:36.042031 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:52:36.041971 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-19a85-kube-rbac-proxy-sar-config\"" Jul 27 17:52:36.049475 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:52:36.049450 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-19a85-bc74c9775-9pdst"] Jul 27 17:52:36.148377 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:52:36.148326 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/935abf93-63f4-43b9-9375-dc83dac68b0a-openshift-service-ca-bundle\") pod \"sequence-graph-19a85-bc74c9775-9pdst\" (UID: \"935abf93-63f4-43b9-9375-dc83dac68b0a\") " pod="kserve-ci-e2e-test/sequence-graph-19a85-bc74c9775-9pdst" Jul 27 17:52:36.148377 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:52:36.148372 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/935abf93-63f4-43b9-9375-dc83dac68b0a-proxy-tls\") pod \"sequence-graph-19a85-bc74c9775-9pdst\" (UID: \"935abf93-63f4-43b9-9375-dc83dac68b0a\") " pod="kserve-ci-e2e-test/sequence-graph-19a85-bc74c9775-9pdst" Jul 27 17:52:36.249230 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:52:36.249188 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/935abf93-63f4-43b9-9375-dc83dac68b0a-openshift-service-ca-bundle\") pod \"sequence-graph-19a85-bc74c9775-9pdst\" (UID: \"935abf93-63f4-43b9-9375-dc83dac68b0a\") " pod="kserve-ci-e2e-test/sequence-graph-19a85-bc74c9775-9pdst" Jul 27 17:52:36.249230 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:52:36.249229 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/935abf93-63f4-43b9-9375-dc83dac68b0a-proxy-tls\") pod \"sequence-graph-19a85-bc74c9775-9pdst\" (UID: \"935abf93-63f4-43b9-9375-dc83dac68b0a\") " pod="kserve-ci-e2e-test/sequence-graph-19a85-bc74c9775-9pdst" Jul 27 17:52:36.250007 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:52:36.249990 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/935abf93-63f4-43b9-9375-dc83dac68b0a-openshift-service-ca-bundle\") pod \"sequence-graph-19a85-bc74c9775-9pdst\" (UID: \"935abf93-63f4-43b9-9375-dc83dac68b0a\") " pod="kserve-ci-e2e-test/sequence-graph-19a85-bc74c9775-9pdst" Jul 27 17:52:36.251781 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:52:36.251765 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/935abf93-63f4-43b9-9375-dc83dac68b0a-proxy-tls\") pod \"sequence-graph-19a85-bc74c9775-9pdst\" (UID: \"935abf93-63f4-43b9-9375-dc83dac68b0a\") " pod="kserve-ci-e2e-test/sequence-graph-19a85-bc74c9775-9pdst" Jul 27 17:52:36.350197 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:52:36.350109 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-19a85-bc74c9775-9pdst" Jul 27 17:52:36.480729 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:52:36.480693 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-19a85-bc74c9775-9pdst"] Jul 27 17:52:36.488591 ip-10-0-140-185 kubenswrapper[2570]: W0727 17:52:36.488563 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod935abf93_63f4_43b9_9375_dc83dac68b0a.slice/crio-9561b315fa9272dd1e3ef301e9440632a1120a91c9b19c6bb59a5fc8897f566e WatchSource:0}: Error finding container 9561b315fa9272dd1e3ef301e9440632a1120a91c9b19c6bb59a5fc8897f566e: Status 404 returned error can't find the container with id 9561b315fa9272dd1e3ef301e9440632a1120a91c9b19c6bb59a5fc8897f566e Jul 27 17:52:36.986577 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:52:36.986537 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-19a85-bc74c9775-9pdst" event={"ID":"935abf93-63f4-43b9-9375-dc83dac68b0a","Type":"ContainerStarted","Data":"76721a9520b20f3da1edd9747fd3dd3211f3a993da0545cc0f098c7f6415bf08"} Jul 27 17:52:36.986748 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:52:36.986593 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-19a85-bc74c9775-9pdst" event={"ID":"935abf93-63f4-43b9-9375-dc83dac68b0a","Type":"ContainerStarted","Data":"9561b315fa9272dd1e3ef301e9440632a1120a91c9b19c6bb59a5fc8897f566e"} Jul 27 17:52:36.986748 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:52:36.986668 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-19a85-bc74c9775-9pdst" Jul 27 17:52:37.003064 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:52:37.003014 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-19a85-bc74c9775-9pdst" podStartSLOduration=1.002997905 podStartE2EDuration="1.002997905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:52:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:52:37.001774644 +0000 UTC m=+1474.442989618" watchObservedRunningTime="2026-07-27 17:52:37.002997905 +0000 UTC m=+1474.444212878" Jul 27 17:52:42.996421 ip-10-0-140-185 kubenswrapper[2570]: I0727 17:52:42.996397 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-19a85-bc74c9775-9pdst" Jul 27 18:00:23.624764 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:00:23.624734 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-28de7-866ccb47df-82qn8"] Jul 27 18:00:23.625232 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:00:23.624920 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-28de7-866ccb47df-82qn8" podUID="28bcd208-9c5c-4e7b-89e8-efe4a0edd09f" containerName="ensemble-graph-28de7" containerID="cri-o://0985f4701160a52cbc03eec61f2879ee30e916e8d6fb6b2a3e6613f6a046a14e" gracePeriod=30 Jul 27 18:00:27.757950 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:00:27.757908 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-28de7-866ccb47df-82qn8" podUID="28bcd208-9c5c-4e7b-89e8-efe4a0edd09f" containerName="ensemble-graph-28de7" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 18:00:32.757778 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:00:32.757734 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-28de7-866ccb47df-82qn8" podUID="28bcd208-9c5c-4e7b-89e8-efe4a0edd09f" containerName="ensemble-graph-28de7" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 18:00:37.757961 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:00:37.757922 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-28de7-866ccb47df-82qn8" podUID="28bcd208-9c5c-4e7b-89e8-efe4a0edd09f" containerName="ensemble-graph-28de7" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 18:00:37.758360 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:00:37.758032 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-28de7-866ccb47df-82qn8" Jul 27 18:00:42.757710 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:00:42.757666 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-28de7-866ccb47df-82qn8" podUID="28bcd208-9c5c-4e7b-89e8-efe4a0edd09f" containerName="ensemble-graph-28de7" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 18:00:47.757414 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:00:47.757377 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-28de7-866ccb47df-82qn8" podUID="28bcd208-9c5c-4e7b-89e8-efe4a0edd09f" containerName="ensemble-graph-28de7" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 18:00:50.603206 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:00:50.603169 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-19a85-bc74c9775-9pdst"] Jul 27 18:00:50.603687 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:00:50.603380 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-19a85-bc74c9775-9pdst" podUID="935abf93-63f4-43b9-9375-dc83dac68b0a" containerName="sequence-graph-19a85" containerID="cri-o://76721a9520b20f3da1edd9747fd3dd3211f3a993da0545cc0f098c7f6415bf08" gracePeriod=30 Jul 27 18:00:52.757155 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:00:52.757115 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-28de7-866ccb47df-82qn8" podUID="28bcd208-9c5c-4e7b-89e8-efe4a0edd09f" containerName="ensemble-graph-28de7" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 18:00:52.994364 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:00:52.994328 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-19a85-bc74c9775-9pdst" podUID="935abf93-63f4-43b9-9375-dc83dac68b0a" containerName="sequence-graph-19a85" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 18:00:53.764308 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:00:53.764278 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-28de7-866ccb47df-82qn8" Jul 27 18:00:53.889763 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:00:53.889678 2570 generic.go:356] "Generic (PLEG): container finished" podID="28bcd208-9c5c-4e7b-89e8-efe4a0edd09f" containerID="0985f4701160a52cbc03eec61f2879ee30e916e8d6fb6b2a3e6613f6a046a14e" exitCode=0 Jul 27 18:00:53.889763 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:00:53.889725 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-28de7-866ccb47df-82qn8" Jul 27 18:00:53.889763 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:00:53.889749 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-28de7-866ccb47df-82qn8" event={"ID":"28bcd208-9c5c-4e7b-89e8-efe4a0edd09f","Type":"ContainerDied","Data":"0985f4701160a52cbc03eec61f2879ee30e916e8d6fb6b2a3e6613f6a046a14e"} Jul 27 18:00:53.890002 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:00:53.889792 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-28de7-866ccb47df-82qn8" event={"ID":"28bcd208-9c5c-4e7b-89e8-efe4a0edd09f","Type":"ContainerDied","Data":"884620ddaebd28434b7b7a683a6314ab30a70f1c61b6c22cd3477154b7431c9f"} Jul 27 18:00:53.890002 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:00:53.889813 2570 scope.go:117] "RemoveContainer" containerID="0985f4701160a52cbc03eec61f2879ee30e916e8d6fb6b2a3e6613f6a046a14e" Jul 27 18:00:53.897559 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:00:53.897542 2570 scope.go:117] "RemoveContainer" containerID="0985f4701160a52cbc03eec61f2879ee30e916e8d6fb6b2a3e6613f6a046a14e" Jul 27 18:00:53.897841 ip-10-0-140-185 kubenswrapper[2570]: E0727 18:00:53.897821 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0985f4701160a52cbc03eec61f2879ee30e916e8d6fb6b2a3e6613f6a046a14e\": container with ID starting with 0985f4701160a52cbc03eec61f2879ee30e916e8d6fb6b2a3e6613f6a046a14e not found: ID does not exist" containerID="0985f4701160a52cbc03eec61f2879ee30e916e8d6fb6b2a3e6613f6a046a14e" Jul 27 18:00:53.897891 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:00:53.897851 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0985f4701160a52cbc03eec61f2879ee30e916e8d6fb6b2a3e6613f6a046a14e"} err="failed to get container status \"0985f4701160a52cbc03eec61f2879ee30e916e8d6fb6b2a3e6613f6a046a14e\": rpc error: code = NotFound desc = could not find container \"0985f4701160a52cbc03eec61f2879ee30e916e8d6fb6b2a3e6613f6a046a14e\": container with ID starting with 0985f4701160a52cbc03eec61f2879ee30e916e8d6fb6b2a3e6613f6a046a14e not found: ID does not exist" Jul 27 18:00:53.904094 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:00:53.904077 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28bcd208-9c5c-4e7b-89e8-efe4a0edd09f-proxy-tls\") pod \"28bcd208-9c5c-4e7b-89e8-efe4a0edd09f\" (UID: \"28bcd208-9c5c-4e7b-89e8-efe4a0edd09f\") " Jul 27 18:00:53.904176 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:00:53.904124 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28bcd208-9c5c-4e7b-89e8-efe4a0edd09f-openshift-service-ca-bundle\") pod \"28bcd208-9c5c-4e7b-89e8-efe4a0edd09f\" (UID: \"28bcd208-9c5c-4e7b-89e8-efe4a0edd09f\") " Jul 27 18:00:53.904494 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:00:53.904475 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28bcd208-9c5c-4e7b-89e8-efe4a0edd09f-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "28bcd208-9c5c-4e7b-89e8-efe4a0edd09f" (UID: "28bcd208-9c5c-4e7b-89e8-efe4a0edd09f"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 18:00:53.906132 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:00:53.906114 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28bcd208-9c5c-4e7b-89e8-efe4a0edd09f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "28bcd208-9c5c-4e7b-89e8-efe4a0edd09f" (UID: "28bcd208-9c5c-4e7b-89e8-efe4a0edd09f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 18:00:54.005380 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:00:54.005342 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28bcd208-9c5c-4e7b-89e8-efe4a0edd09f-openshift-service-ca-bundle\") on node \"ip-10-0-140-185.ec2.internal\" DevicePath \"\"" Jul 27 18:00:54.005380 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:00:54.005377 2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28bcd208-9c5c-4e7b-89e8-efe4a0edd09f-proxy-tls\") on node \"ip-10-0-140-185.ec2.internal\" DevicePath \"\"" Jul 27 18:00:54.209860 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:00:54.209829 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-28de7-866ccb47df-82qn8"] Jul 27 18:00:54.213796 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:00:54.213776 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-28de7-866ccb47df-82qn8"] Jul 27 18:00:55.128476 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:00:55.128442 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28bcd208-9c5c-4e7b-89e8-efe4a0edd09f" path="/var/lib/kubelet/pods/28bcd208-9c5c-4e7b-89e8-efe4a0edd09f/volumes" Jul 27 18:00:57.995978 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:00:57.995941 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-19a85-bc74c9775-9pdst" podUID="935abf93-63f4-43b9-9375-dc83dac68b0a" containerName="sequence-graph-19a85" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 18:01:02.994361 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:01:02.994317 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-19a85-bc74c9775-9pdst" podUID="935abf93-63f4-43b9-9375-dc83dac68b0a" containerName="sequence-graph-19a85" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 18:01:02.994902 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:01:02.994451 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-19a85-bc74c9775-9pdst" Jul 27 18:01:07.995075 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:01:07.995038 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-19a85-bc74c9775-9pdst" podUID="935abf93-63f4-43b9-9375-dc83dac68b0a" containerName="sequence-graph-19a85" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 18:01:12.994189 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:01:12.994153 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-19a85-bc74c9775-9pdst" podUID="935abf93-63f4-43b9-9375-dc83dac68b0a" containerName="sequence-graph-19a85" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 18:01:17.994773 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:01:17.994736 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-19a85-bc74c9775-9pdst" podUID="935abf93-63f4-43b9-9375-dc83dac68b0a" containerName="sequence-graph-19a85" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 18:01:20.747503 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:01:20.747476 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-19a85-bc74c9775-9pdst" Jul 27 18:01:20.818241 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:01:20.818210 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/935abf93-63f4-43b9-9375-dc83dac68b0a-openshift-service-ca-bundle\") pod \"935abf93-63f4-43b9-9375-dc83dac68b0a\" (UID: \"935abf93-63f4-43b9-9375-dc83dac68b0a\") " Jul 27 18:01:20.818395 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:01:20.818316 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/935abf93-63f4-43b9-9375-dc83dac68b0a-proxy-tls\") pod \"935abf93-63f4-43b9-9375-dc83dac68b0a\" (UID: \"935abf93-63f4-43b9-9375-dc83dac68b0a\") " Jul 27 18:01:20.818647 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:01:20.818626 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/935abf93-63f4-43b9-9375-dc83dac68b0a-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "935abf93-63f4-43b9-9375-dc83dac68b0a" (UID: "935abf93-63f4-43b9-9375-dc83dac68b0a"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 18:01:20.820311 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:01:20.820291 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/935abf93-63f4-43b9-9375-dc83dac68b0a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "935abf93-63f4-43b9-9375-dc83dac68b0a" (UID: "935abf93-63f4-43b9-9375-dc83dac68b0a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 18:01:20.919678 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:01:20.919601 2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/935abf93-63f4-43b9-9375-dc83dac68b0a-proxy-tls\") on node \"ip-10-0-140-185.ec2.internal\" DevicePath \"\"" Jul 27 18:01:20.919678 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:01:20.919631 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/935abf93-63f4-43b9-9375-dc83dac68b0a-openshift-service-ca-bundle\") on node \"ip-10-0-140-185.ec2.internal\" DevicePath \"\"" Jul 27 18:01:20.999650 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:01:20.999608 2570 generic.go:356] "Generic (PLEG): container finished" podID="935abf93-63f4-43b9-9375-dc83dac68b0a" containerID="76721a9520b20f3da1edd9747fd3dd3211f3a993da0545cc0f098c7f6415bf08" exitCode=0 Jul 27 18:01:20.999840 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:01:20.999671 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-19a85-bc74c9775-9pdst" Jul 27 18:01:20.999840 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:01:20.999693 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-19a85-bc74c9775-9pdst" event={"ID":"935abf93-63f4-43b9-9375-dc83dac68b0a","Type":"ContainerDied","Data":"76721a9520b20f3da1edd9747fd3dd3211f3a993da0545cc0f098c7f6415bf08"} Jul 27 18:01:20.999840 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:01:20.999769 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-19a85-bc74c9775-9pdst" event={"ID":"935abf93-63f4-43b9-9375-dc83dac68b0a","Type":"ContainerDied","Data":"9561b315fa9272dd1e3ef301e9440632a1120a91c9b19c6bb59a5fc8897f566e"} Jul 27 18:01:20.999840 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:01:20.999791 2570 scope.go:117] "RemoveContainer" containerID="76721a9520b20f3da1edd9747fd3dd3211f3a993da0545cc0f098c7f6415bf08" Jul 27 18:01:21.008139 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:01:21.008122 2570 scope.go:117] "RemoveContainer" containerID="76721a9520b20f3da1edd9747fd3dd3211f3a993da0545cc0f098c7f6415bf08" Jul 27 18:01:21.008386 ip-10-0-140-185 kubenswrapper[2570]: E0727 18:01:21.008370 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76721a9520b20f3da1edd9747fd3dd3211f3a993da0545cc0f098c7f6415bf08\": container with ID starting with 76721a9520b20f3da1edd9747fd3dd3211f3a993da0545cc0f098c7f6415bf08 not found: ID does not exist" containerID="76721a9520b20f3da1edd9747fd3dd3211f3a993da0545cc0f098c7f6415bf08" Jul 27 18:01:21.008432 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:01:21.008394 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76721a9520b20f3da1edd9747fd3dd3211f3a993da0545cc0f098c7f6415bf08"} err="failed to get container status \"76721a9520b20f3da1edd9747fd3dd3211f3a993da0545cc0f098c7f6415bf08\": rpc error: code = NotFound desc = could not find container \"76721a9520b20f3da1edd9747fd3dd3211f3a993da0545cc0f098c7f6415bf08\": container with ID starting with 76721a9520b20f3da1edd9747fd3dd3211f3a993da0545cc0f098c7f6415bf08 not found: ID does not exist" Jul 27 18:01:21.019834 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:01:21.019808 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-19a85-bc74c9775-9pdst"] Jul 27 18:01:21.023561 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:01:21.023540 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-19a85-bc74c9775-9pdst"] Jul 27 18:01:21.128684 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:01:21.128655 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="935abf93-63f4-43b9-9375-dc83dac68b0a" path="/var/lib/kubelet/pods/935abf93-63f4-43b9-9375-dc83dac68b0a/volumes" Jul 27 18:01:50.878702 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:01:50.878667 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-0a977-76cfd865ff-89slq"] Jul 27 18:01:50.879189 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:01:50.879003 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28bcd208-9c5c-4e7b-89e8-efe4a0edd09f" containerName="ensemble-graph-28de7" Jul 27 18:01:50.879189 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:01:50.879022 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="28bcd208-9c5c-4e7b-89e8-efe4a0edd09f" containerName="ensemble-graph-28de7" Jul 27 18:01:50.879189 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:01:50.879035 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="935abf93-63f4-43b9-9375-dc83dac68b0a" containerName="sequence-graph-19a85" Jul 27 18:01:50.879189 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:01:50.879044 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="935abf93-63f4-43b9-9375-dc83dac68b0a" containerName="sequence-graph-19a85" Jul 27 18:01:50.879189 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:01:50.879122 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="935abf93-63f4-43b9-9375-dc83dac68b0a" containerName="sequence-graph-19a85" Jul 27 18:01:50.879189 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:01:50.879131 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="28bcd208-9c5c-4e7b-89e8-efe4a0edd09f" containerName="ensemble-graph-28de7" Jul 27 18:01:50.882012 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:01:50.881995 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-0a977-76cfd865ff-89slq" Jul 27 18:01:50.884207 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:01:50.884184 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-0a977-kube-rbac-proxy-sar-config\"" Jul 27 18:01:50.884311 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:01:50.884213 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 27 18:01:50.884311 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:01:50.884189 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-x6c7x\"" Jul 27 18:01:50.884510 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:01:50.884488 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-0a977-serving-cert\"" Jul 27 18:01:50.891651 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:01:50.891631 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-0a977-76cfd865ff-89slq"] Jul 27 18:01:50.981404 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:01:50.981368 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/990e9e29-4e39-4b21-9bbf-5b60ab26f8f5-openshift-service-ca-bundle\") pod \"switch-graph-0a977-76cfd865ff-89slq\" (UID: \"990e9e29-4e39-4b21-9bbf-5b60ab26f8f5\") " pod="kserve-ci-e2e-test/switch-graph-0a977-76cfd865ff-89slq" Jul 27 18:01:50.981624 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:01:50.981426 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/990e9e29-4e39-4b21-9bbf-5b60ab26f8f5-proxy-tls\") pod \"switch-graph-0a977-76cfd865ff-89slq\" (UID: \"990e9e29-4e39-4b21-9bbf-5b60ab26f8f5\") " pod="kserve-ci-e2e-test/switch-graph-0a977-76cfd865ff-89slq" Jul 27 18:01:51.082364 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:01:51.082336 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/990e9e29-4e39-4b21-9bbf-5b60ab26f8f5-proxy-tls\") pod \"switch-graph-0a977-76cfd865ff-89slq\" (UID: \"990e9e29-4e39-4b21-9bbf-5b60ab26f8f5\") " pod="kserve-ci-e2e-test/switch-graph-0a977-76cfd865ff-89slq" Jul 27 18:01:51.082540 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:01:51.082396 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/990e9e29-4e39-4b21-9bbf-5b60ab26f8f5-openshift-service-ca-bundle\") pod \"switch-graph-0a977-76cfd865ff-89slq\" (UID: \"990e9e29-4e39-4b21-9bbf-5b60ab26f8f5\") " pod="kserve-ci-e2e-test/switch-graph-0a977-76cfd865ff-89slq" Jul 27 18:01:51.082540 ip-10-0-140-185 kubenswrapper[2570]: E0727 18:01:51.082484 2570 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-0a977-serving-cert: secret "switch-graph-0a977-serving-cert" not found Jul 27 18:01:51.082629 ip-10-0-140-185 kubenswrapper[2570]: E0727 18:01:51.082572 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/990e9e29-4e39-4b21-9bbf-5b60ab26f8f5-proxy-tls podName:990e9e29-4e39-4b21-9bbf-5b60ab26f8f5 nodeName:}" failed. No retries permitted until 2026-07-27 18:01:51.58255615 +0000 UTC m=+2029.023771102 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/990e9e29-4e39-4b21-9bbf-5b60ab26f8f5-proxy-tls") pod "switch-graph-0a977-76cfd865ff-89slq" (UID: "990e9e29-4e39-4b21-9bbf-5b60ab26f8f5") : secret "switch-graph-0a977-serving-cert" not found Jul 27 18:01:51.082959 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:01:51.082942 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/990e9e29-4e39-4b21-9bbf-5b60ab26f8f5-openshift-service-ca-bundle\") pod \"switch-graph-0a977-76cfd865ff-89slq\" (UID: \"990e9e29-4e39-4b21-9bbf-5b60ab26f8f5\") " pod="kserve-ci-e2e-test/switch-graph-0a977-76cfd865ff-89slq" Jul 27 18:01:51.585921 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:01:51.585886 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/990e9e29-4e39-4b21-9bbf-5b60ab26f8f5-proxy-tls\") pod \"switch-graph-0a977-76cfd865ff-89slq\" (UID: \"990e9e29-4e39-4b21-9bbf-5b60ab26f8f5\") " pod="kserve-ci-e2e-test/switch-graph-0a977-76cfd865ff-89slq" Jul 27 18:01:51.588338 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:01:51.588315 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/990e9e29-4e39-4b21-9bbf-5b60ab26f8f5-proxy-tls\") pod \"switch-graph-0a977-76cfd865ff-89slq\" (UID: \"990e9e29-4e39-4b21-9bbf-5b60ab26f8f5\") " pod="kserve-ci-e2e-test/switch-graph-0a977-76cfd865ff-89slq" Jul 27 18:01:51.792006 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:01:51.791970 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-0a977-76cfd865ff-89slq" Jul 27 18:01:51.914034 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:01:51.914002 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-0a977-76cfd865ff-89slq"] Jul 27 18:01:51.917172 ip-10-0-140-185 kubenswrapper[2570]: W0727 18:01:51.917141 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod990e9e29_4e39_4b21_9bbf_5b60ab26f8f5.slice/crio-b378985daad8ab83f6389027b72ba225f91c943c4ac4df8cd40c02b6d93806c4 WatchSource:0}: Error finding container b378985daad8ab83f6389027b72ba225f91c943c4ac4df8cd40c02b6d93806c4: Status 404 returned error can't find the container with id b378985daad8ab83f6389027b72ba225f91c943c4ac4df8cd40c02b6d93806c4 Jul 27 18:01:52.116862 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:01:52.116779 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-0a977-76cfd865ff-89slq" event={"ID":"990e9e29-4e39-4b21-9bbf-5b60ab26f8f5","Type":"ContainerStarted","Data":"89585c6d40a184b1fbdf1fedc11dbc64f68eff40af8d909284016dc8db214f8c"} Jul 27 18:01:52.116862 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:01:52.116834 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-0a977-76cfd865ff-89slq" event={"ID":"990e9e29-4e39-4b21-9bbf-5b60ab26f8f5","Type":"ContainerStarted","Data":"b378985daad8ab83f6389027b72ba225f91c943c4ac4df8cd40c02b6d93806c4"} Jul 27 18:01:52.117054 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:01:52.116895 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-0a977-76cfd865ff-89slq" Jul 27 18:01:52.132913 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:01:52.132855 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-0a977-76cfd865ff-89slq" podStartSLOduration=2.132838168 podStartE2EDuration="2.132838168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:01:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:01:52.131120865 +0000 UTC m=+2029.572335841" watchObservedRunningTime="2026-07-27 18:01:52.132838168 +0000 UTC m=+2029.574053144" Jul 27 18:01:58.126049 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:01:58.126022 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-0a977-76cfd865ff-89slq" Jul 27 18:18:10.200507 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:10.200475 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-0a977-76cfd865ff-89slq"] Jul 27 18:18:10.203018 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:10.200707 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-0a977-76cfd865ff-89slq" podUID="990e9e29-4e39-4b21-9bbf-5b60ab26f8f5" containerName="switch-graph-0a977" containerID="cri-o://89585c6d40a184b1fbdf1fedc11dbc64f68eff40af8d909284016dc8db214f8c" gracePeriod=30 Jul 27 18:18:13.125206 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:13.125157 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-0a977-76cfd865ff-89slq" podUID="990e9e29-4e39-4b21-9bbf-5b60ab26f8f5" containerName="switch-graph-0a977" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 18:18:18.124676 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:18.124637 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-0a977-76cfd865ff-89slq" podUID="990e9e29-4e39-4b21-9bbf-5b60ab26f8f5" containerName="switch-graph-0a977" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 18:18:23.124825 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:23.124770 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-0a977-76cfd865ff-89slq" podUID="990e9e29-4e39-4b21-9bbf-5b60ab26f8f5" containerName="switch-graph-0a977" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 18:18:23.129103 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:23.129081 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-0a977-76cfd865ff-89slq" Jul 27 18:18:27.596408 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:27.596384 2570 ???:1] "http: TLS handshake error from 10.0.140.185:58064: EOF" Jul 27 18:18:27.599903 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:27.599876 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-0a977-76cfd865ff-89slq_990e9e29-4e39-4b21-9bbf-5b60ab26f8f5/switch-graph-0a977/0.log" Jul 27 18:18:28.124531 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:28.124484 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-0a977-76cfd865ff-89slq" podUID="990e9e29-4e39-4b21-9bbf-5b60ab26f8f5" containerName="switch-graph-0a977" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 18:18:28.352208 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:28.352179 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-0a977-76cfd865ff-89slq_990e9e29-4e39-4b21-9bbf-5b60ab26f8f5/switch-graph-0a977/0.log" Jul 27 18:18:29.174018 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:29.173992 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-0a977-76cfd865ff-89slq_990e9e29-4e39-4b21-9bbf-5b60ab26f8f5/switch-graph-0a977/0.log" Jul 27 18:18:29.906922 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:29.906891 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-0a977-76cfd865ff-89slq_990e9e29-4e39-4b21-9bbf-5b60ab26f8f5/switch-graph-0a977/0.log" Jul 27 18:18:30.642653 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:30.642620 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-0a977-76cfd865ff-89slq_990e9e29-4e39-4b21-9bbf-5b60ab26f8f5/switch-graph-0a977/0.log" Jul 27 18:18:31.395489 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:31.395441 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-0a977-76cfd865ff-89slq_990e9e29-4e39-4b21-9bbf-5b60ab26f8f5/switch-graph-0a977/0.log" Jul 27 18:18:32.145405 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:32.145379 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-0a977-76cfd865ff-89slq_990e9e29-4e39-4b21-9bbf-5b60ab26f8f5/switch-graph-0a977/0.log" Jul 27 18:18:32.939406 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:32.939378 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-0a977-76cfd865ff-89slq_990e9e29-4e39-4b21-9bbf-5b60ab26f8f5/switch-graph-0a977/0.log" Jul 27 18:18:33.124597 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:33.124558 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-0a977-76cfd865ff-89slq" podUID="990e9e29-4e39-4b21-9bbf-5b60ab26f8f5" containerName="switch-graph-0a977" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 18:18:33.673047 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:33.673018 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-0a977-76cfd865ff-89slq_990e9e29-4e39-4b21-9bbf-5b60ab26f8f5/switch-graph-0a977/0.log" Jul 27 18:18:34.436533 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:34.436482 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-0a977-76cfd865ff-89slq_990e9e29-4e39-4b21-9bbf-5b60ab26f8f5/switch-graph-0a977/0.log" Jul 27 18:18:35.192513 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:35.192483 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-0a977-76cfd865ff-89slq_990e9e29-4e39-4b21-9bbf-5b60ab26f8f5/switch-graph-0a977/0.log" Jul 27 18:18:35.932966 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:35.932936 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-0a977-76cfd865ff-89slq_990e9e29-4e39-4b21-9bbf-5b60ab26f8f5/switch-graph-0a977/0.log" Jul 27 18:18:36.676388 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:36.676360 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-0a977-76cfd865ff-89slq_990e9e29-4e39-4b21-9bbf-5b60ab26f8f5/switch-graph-0a977/0.log" Jul 27 18:18:37.417706 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:37.417674 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-0a977-76cfd865ff-89slq_990e9e29-4e39-4b21-9bbf-5b60ab26f8f5/switch-graph-0a977/0.log" Jul 27 18:18:38.124428 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:38.124391 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-0a977-76cfd865ff-89slq" podUID="990e9e29-4e39-4b21-9bbf-5b60ab26f8f5" containerName="switch-graph-0a977" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 18:18:40.349036 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:40.349011 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-0a977-76cfd865ff-89slq" Jul 27 18:18:40.409008 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:40.408981 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/990e9e29-4e39-4b21-9bbf-5b60ab26f8f5-openshift-service-ca-bundle\") pod \"990e9e29-4e39-4b21-9bbf-5b60ab26f8f5\" (UID: \"990e9e29-4e39-4b21-9bbf-5b60ab26f8f5\") " Jul 27 18:18:40.409196 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:40.409023 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/990e9e29-4e39-4b21-9bbf-5b60ab26f8f5-proxy-tls\") pod \"990e9e29-4e39-4b21-9bbf-5b60ab26f8f5\" (UID: \"990e9e29-4e39-4b21-9bbf-5b60ab26f8f5\") " Jul 27 18:18:40.409386 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:40.409362 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/990e9e29-4e39-4b21-9bbf-5b60ab26f8f5-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "990e9e29-4e39-4b21-9bbf-5b60ab26f8f5" (UID: "990e9e29-4e39-4b21-9bbf-5b60ab26f8f5"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 18:18:40.411173 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:40.411149 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/990e9e29-4e39-4b21-9bbf-5b60ab26f8f5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "990e9e29-4e39-4b21-9bbf-5b60ab26f8f5" (UID: "990e9e29-4e39-4b21-9bbf-5b60ab26f8f5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 18:18:40.509919 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:40.509896 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/990e9e29-4e39-4b21-9bbf-5b60ab26f8f5-openshift-service-ca-bundle\") on node \"ip-10-0-140-185.ec2.internal\" DevicePath \"\"" Jul 27 18:18:40.509919 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:40.509922 2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/990e9e29-4e39-4b21-9bbf-5b60ab26f8f5-proxy-tls\") on node \"ip-10-0-140-185.ec2.internal\" DevicePath \"\"" Jul 27 18:18:40.975691 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:40.975606 2570 generic.go:356] "Generic (PLEG): container finished" podID="990e9e29-4e39-4b21-9bbf-5b60ab26f8f5" containerID="89585c6d40a184b1fbdf1fedc11dbc64f68eff40af8d909284016dc8db214f8c" exitCode=0 Jul 27 18:18:40.975691 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:40.975647 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-0a977-76cfd865ff-89slq" Jul 27 18:18:40.975863 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:40.975651 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-0a977-76cfd865ff-89slq" event={"ID":"990e9e29-4e39-4b21-9bbf-5b60ab26f8f5","Type":"ContainerDied","Data":"89585c6d40a184b1fbdf1fedc11dbc64f68eff40af8d909284016dc8db214f8c"} Jul 27 18:18:40.975863 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:40.975753 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-0a977-76cfd865ff-89slq" event={"ID":"990e9e29-4e39-4b21-9bbf-5b60ab26f8f5","Type":"ContainerDied","Data":"b378985daad8ab83f6389027b72ba225f91c943c4ac4df8cd40c02b6d93806c4"} Jul 27 18:18:40.975863 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:40.975776 2570 scope.go:117] "RemoveContainer" containerID="89585c6d40a184b1fbdf1fedc11dbc64f68eff40af8d909284016dc8db214f8c" Jul 27 18:18:40.986588 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:40.986569 2570 scope.go:117] "RemoveContainer" containerID="89585c6d40a184b1fbdf1fedc11dbc64f68eff40af8d909284016dc8db214f8c" Jul 27 18:18:40.986884 ip-10-0-140-185 kubenswrapper[2570]: E0727 18:18:40.986860 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89585c6d40a184b1fbdf1fedc11dbc64f68eff40af8d909284016dc8db214f8c\": container with ID starting with 89585c6d40a184b1fbdf1fedc11dbc64f68eff40af8d909284016dc8db214f8c not found: ID does not exist" containerID="89585c6d40a184b1fbdf1fedc11dbc64f68eff40af8d909284016dc8db214f8c" Jul 27 18:18:40.986960 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:40.986895 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89585c6d40a184b1fbdf1fedc11dbc64f68eff40af8d909284016dc8db214f8c"} err="failed to get container status \"89585c6d40a184b1fbdf1fedc11dbc64f68eff40af8d909284016dc8db214f8c\": rpc error: code = NotFound desc = could not find container \"89585c6d40a184b1fbdf1fedc11dbc64f68eff40af8d909284016dc8db214f8c\": container with ID starting with 89585c6d40a184b1fbdf1fedc11dbc64f68eff40af8d909284016dc8db214f8c not found: ID does not exist" Jul 27 18:18:40.997349 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:40.997324 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-0a977-76cfd865ff-89slq"] Jul 27 18:18:41.000943 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:41.000922 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-0a977-76cfd865ff-89slq"] Jul 27 18:18:41.128569 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:41.128537 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="990e9e29-4e39-4b21-9bbf-5b60ab26f8f5" path="/var/lib/kubelet/pods/990e9e29-4e39-4b21-9bbf-5b60ab26f8f5/volumes" Jul 27 18:18:46.449534 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:46.449502 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-d4nqw_7209ee7f-3019-4465-8175-3317741f9b09/global-pull-secret-syncer/0.log" Jul 27 18:18:46.656946 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:46.656901 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-l5bjn_0e66277b-cc74-421a-84d7-d1d43ce8de57/konnectivity-agent/0.log" Jul 27 18:18:46.745108 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:46.745030 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-185.ec2.internal_92aa1923735291b61478f523669c7af8/haproxy/0.log" Jul 27 18:18:50.924509 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:50.924481 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_afd26584-d3b3-434a-9f1a-a4db8f53259f/alertmanager/0.log" Jul 27 18:18:50.955286 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:50.955243 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_afd26584-d3b3-434a-9f1a-a4db8f53259f/config-reloader/0.log" Jul 27 18:18:50.982601 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:50.982575 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_afd26584-d3b3-434a-9f1a-a4db8f53259f/kube-rbac-proxy-web/0.log" Jul 27 18:18:51.005079 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:51.005026 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_afd26584-d3b3-434a-9f1a-a4db8f53259f/kube-rbac-proxy/0.log" Jul 27 18:18:51.027727 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:51.027685 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_afd26584-d3b3-434a-9f1a-a4db8f53259f/kube-rbac-proxy-metric/0.log" Jul 27 18:18:51.052778 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:51.052759 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_afd26584-d3b3-434a-9f1a-a4db8f53259f/prom-label-proxy/0.log" Jul 27 18:18:51.075558 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:51.075537 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_afd26584-d3b3-434a-9f1a-a4db8f53259f/init-config-reloader/0.log" Jul 27 18:18:51.158798 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:51.158771 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-wc86b_9e2569a9-2352-4e32-96b1-fc35089c290e/kube-state-metrics/0.log" Jul 27 18:18:51.182160 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:51.182095 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-wc86b_9e2569a9-2352-4e32-96b1-fc35089c290e/kube-rbac-proxy-main/0.log" Jul 27 18:18:51.205083 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:51.205053 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-wc86b_9e2569a9-2352-4e32-96b1-fc35089c290e/kube-rbac-proxy-self/0.log" Jul 27 18:18:51.274487 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:51.274456 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-gbk2t_d1faee83-1b22-4bb7-b51d-06f34785dc23/monitoring-plugin/0.log" Jul 27 18:18:51.375163 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:51.375132 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h67d4_6269d5fd-1774-4aef-9cc7-2296db21f5a3/node-exporter/0.log" Jul 27 18:18:51.404843 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:51.404809 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h67d4_6269d5fd-1774-4aef-9cc7-2296db21f5a3/kube-rbac-proxy/0.log" Jul 27 18:18:51.434080 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:51.434012 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h67d4_6269d5fd-1774-4aef-9cc7-2296db21f5a3/init-textfile/0.log" Jul 27 18:18:51.551283 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:51.551260 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-cckjt_a0aee75d-e83d-4d0e-a70e-edea165b58c7/kube-rbac-proxy-main/0.log" Jul 27 18:18:51.574557 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:51.574533 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-cckjt_a0aee75d-e83d-4d0e-a70e-edea165b58c7/kube-rbac-proxy-self/0.log" Jul 27 18:18:51.604568 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:51.604542 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-cckjt_a0aee75d-e83d-4d0e-a70e-edea165b58c7/openshift-state-metrics/0.log" Jul 27 18:18:51.913650 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:51.913616 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-n974k_c0b6a6c4-65f9-43b0-b176-aa722c1504df/prometheus-operator-admission-webhook/0.log" Jul 27 18:18:51.944535 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:51.944479 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-549d7665ff-sth4s_4378d963-6214-464e-80f6-89f704ebe2f4/telemeter-client/0.log" Jul 27 18:18:51.969656 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:51.969624 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-549d7665ff-sth4s_4378d963-6214-464e-80f6-89f704ebe2f4/reload/0.log" Jul 27 18:18:51.993780 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:51.993756 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-549d7665ff-sth4s_4378d963-6214-464e-80f6-89f704ebe2f4/kube-rbac-proxy/0.log" Jul 27 18:18:53.491389 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:53.491361 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-g9grw_f60c07b8-db05-40f2-b0ac-64df9e16e9e9/networking-console-plugin/0.log" Jul 27 18:18:54.473784 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:54.473697 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-bgt4v_ad4401c9-a50e-46d6-a1f6-e4d8bf591064/download-server/0.log" Jul 27 18:18:55.076758 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:55.076726 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mlzzr/perf-node-gather-daemonset-n6zhz"] Jul 27 18:18:55.077115 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:55.077068 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="990e9e29-4e39-4b21-9bbf-5b60ab26f8f5" containerName="switch-graph-0a977" Jul 27 18:18:55.077115 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:55.077079 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="990e9e29-4e39-4b21-9bbf-5b60ab26f8f5" containerName="switch-graph-0a977" Jul 27 18:18:55.077184 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:55.077131 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="990e9e29-4e39-4b21-9bbf-5b60ab26f8f5" containerName="switch-graph-0a977" Jul 27 18:18:55.082043 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:55.082014 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mlzzr/perf-node-gather-daemonset-n6zhz" Jul 27 18:18:55.084640 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:55.084614 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mlzzr\"/\"openshift-service-ca.crt\"" Jul 27 18:18:55.084783 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:55.084651 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mlzzr\"/\"default-dockercfg-hfqqn\"" Jul 27 18:18:55.085287 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:55.085267 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mlzzr\"/\"kube-root-ca.crt\"" Jul 27 18:18:55.089257 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:55.089176 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mlzzr/perf-node-gather-daemonset-n6zhz"] Jul 27 18:18:55.231645 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:55.231611 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/deac2e40-c990-4878-995b-c81e375222d3-sys\") pod \"perf-node-gather-daemonset-n6zhz\" (UID: \"deac2e40-c990-4878-995b-c81e375222d3\") " pod="openshift-must-gather-mlzzr/perf-node-gather-daemonset-n6zhz" Jul 27 18:18:55.231823 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:55.231691 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/deac2e40-c990-4878-995b-c81e375222d3-lib-modules\") pod \"perf-node-gather-daemonset-n6zhz\" (UID: \"deac2e40-c990-4878-995b-c81e375222d3\") " pod="openshift-must-gather-mlzzr/perf-node-gather-daemonset-n6zhz" Jul 27 18:18:55.231823 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:55.231745 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/deac2e40-c990-4878-995b-c81e375222d3-podres\") pod \"perf-node-gather-daemonset-n6zhz\" (UID: \"deac2e40-c990-4878-995b-c81e375222d3\") " pod="openshift-must-gather-mlzzr/perf-node-gather-daemonset-n6zhz" Jul 27 18:18:55.231906 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:55.231815 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xmpl\" (UniqueName: \"kubernetes.io/projected/deac2e40-c990-4878-995b-c81e375222d3-kube-api-access-2xmpl\") pod \"perf-node-gather-daemonset-n6zhz\" (UID: \"deac2e40-c990-4878-995b-c81e375222d3\") " pod="openshift-must-gather-mlzzr/perf-node-gather-daemonset-n6zhz" Jul 27 18:18:55.231985 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:55.231958 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/deac2e40-c990-4878-995b-c81e375222d3-proc\") pod \"perf-node-gather-daemonset-n6zhz\" (UID: \"deac2e40-c990-4878-995b-c81e375222d3\") " pod="openshift-must-gather-mlzzr/perf-node-gather-daemonset-n6zhz" Jul 27 18:18:55.332577 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:55.332495 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xmpl\" (UniqueName: \"kubernetes.io/projected/deac2e40-c990-4878-995b-c81e375222d3-kube-api-access-2xmpl\") pod \"perf-node-gather-daemonset-n6zhz\" (UID: \"deac2e40-c990-4878-995b-c81e375222d3\") " pod="openshift-must-gather-mlzzr/perf-node-gather-daemonset-n6zhz" Jul 27 18:18:55.332700 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:55.332576 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/deac2e40-c990-4878-995b-c81e375222d3-proc\") pod \"perf-node-gather-daemonset-n6zhz\" (UID: \"deac2e40-c990-4878-995b-c81e375222d3\") " pod="openshift-must-gather-mlzzr/perf-node-gather-daemonset-n6zhz" Jul 27 18:18:55.332700 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:55.332628 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/deac2e40-c990-4878-995b-c81e375222d3-sys\") pod \"perf-node-gather-daemonset-n6zhz\" (UID: \"deac2e40-c990-4878-995b-c81e375222d3\") " pod="openshift-must-gather-mlzzr/perf-node-gather-daemonset-n6zhz" Jul 27 18:18:55.332700 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:55.332678 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/deac2e40-c990-4878-995b-c81e375222d3-sys\") pod \"perf-node-gather-daemonset-n6zhz\" (UID: \"deac2e40-c990-4878-995b-c81e375222d3\") " pod="openshift-must-gather-mlzzr/perf-node-gather-daemonset-n6zhz" Jul 27 18:18:55.332700 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:55.332680 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/deac2e40-c990-4878-995b-c81e375222d3-proc\") pod \"perf-node-gather-daemonset-n6zhz\" (UID: \"deac2e40-c990-4878-995b-c81e375222d3\") " pod="openshift-must-gather-mlzzr/perf-node-gather-daemonset-n6zhz" Jul 27 18:18:55.332864 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:55.332725 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/deac2e40-c990-4878-995b-c81e375222d3-lib-modules\") pod \"perf-node-gather-daemonset-n6zhz\" (UID: \"deac2e40-c990-4878-995b-c81e375222d3\") " pod="openshift-must-gather-mlzzr/perf-node-gather-daemonset-n6zhz" Jul 27 18:18:55.332864 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:55.332770 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/deac2e40-c990-4878-995b-c81e375222d3-podres\") pod \"perf-node-gather-daemonset-n6zhz\" (UID: \"deac2e40-c990-4878-995b-c81e375222d3\") " pod="openshift-must-gather-mlzzr/perf-node-gather-daemonset-n6zhz" Jul 27 18:18:55.332935 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:55.332889 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/deac2e40-c990-4878-995b-c81e375222d3-lib-modules\") pod \"perf-node-gather-daemonset-n6zhz\" (UID: \"deac2e40-c990-4878-995b-c81e375222d3\") " pod="openshift-must-gather-mlzzr/perf-node-gather-daemonset-n6zhz" Jul 27 18:18:55.332935 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:55.332891 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/deac2e40-c990-4878-995b-c81e375222d3-podres\") pod \"perf-node-gather-daemonset-n6zhz\" (UID: \"deac2e40-c990-4878-995b-c81e375222d3\") " pod="openshift-must-gather-mlzzr/perf-node-gather-daemonset-n6zhz" Jul 27 18:18:55.339180 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:55.339159 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xmpl\" (UniqueName: \"kubernetes.io/projected/deac2e40-c990-4878-995b-c81e375222d3-kube-api-access-2xmpl\") pod \"perf-node-gather-daemonset-n6zhz\" (UID: \"deac2e40-c990-4878-995b-c81e375222d3\") " pod="openshift-must-gather-mlzzr/perf-node-gather-daemonset-n6zhz" Jul 27 18:18:55.393349 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:55.393315 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mlzzr/perf-node-gather-daemonset-n6zhz" Jul 27 18:18:55.518601 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:55.518483 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mlzzr/perf-node-gather-daemonset-n6zhz"] Jul 27 18:18:55.521190 ip-10-0-140-185 kubenswrapper[2570]: W0727 18:18:55.521166 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poddeac2e40_c990_4878_995b_c81e375222d3.slice/crio-28751d5dcb17c2756c6281f963efc0736a055bcf7851ff00e22c33c85ddd14b3 WatchSource:0}: Error finding container 28751d5dcb17c2756c6281f963efc0736a055bcf7851ff00e22c33c85ddd14b3: Status 404 returned error can't find the container with id 28751d5dcb17c2756c6281f963efc0736a055bcf7851ff00e22c33c85ddd14b3 Jul 27 18:18:55.926178 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:55.926102 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dpkfg_91b9ae5b-5c22-41ab-8845-cd9f65146cbc/dns/0.log" Jul 27 18:18:55.950171 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:55.950136 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dpkfg_91b9ae5b-5c22-41ab-8845-cd9f65146cbc/kube-rbac-proxy/0.log" Jul 27 18:18:56.039913 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:56.039877 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mlzzr/perf-node-gather-daemonset-n6zhz" event={"ID":"deac2e40-c990-4878-995b-c81e375222d3","Type":"ContainerStarted","Data":"45c12f7aa140ef9434fdc06c7bd14899416decf240114cafb9b0965053901f38"} Jul 27 18:18:56.040090 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:56.039930 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mlzzr/perf-node-gather-daemonset-n6zhz" event={"ID":"deac2e40-c990-4878-995b-c81e375222d3","Type":"ContainerStarted","Data":"28751d5dcb17c2756c6281f963efc0736a055bcf7851ff00e22c33c85ddd14b3"} Jul 27 18:18:56.040090 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:56.040017 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mlzzr/perf-node-gather-daemonset-n6zhz" Jul 27 18:18:56.054127 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:56.054085 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mlzzr/perf-node-gather-daemonset-n6zhz" podStartSLOduration=1.054069938 podStartE2EDuration="1.054069938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 18:18:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:18:56.05322285 +0000 UTC m=+3053.494437835" watchObservedRunningTime="2026-07-27 18:18:56.054069938 +0000 UTC m=+3053.495284912" Jul 27 18:18:56.087588 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:56.087563 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-f22lz_b8d2f0e3-917e-4be9-adf5-e233e4f97048/dns-node-resolver/0.log" Jul 27 18:18:56.595721 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:56.595687 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6bccbd7d69-hmc99_cf05ff2e-7a6f-4f26-aedc-302f49afdd34/registry/0.log" Jul 27 18:18:56.620297 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:56.620275 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-f5x5s_acba0339-7147-4770-8e0b-ae7fff7eba7e/node-ca/0.log" Jul 27 18:18:57.738248 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:57.738215 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gc5j9_1596f129-a503-4513-bb1f-3b7ac6339c51/serve-healthcheck-canary/0.log" Jul 27 18:18:58.279235 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:58.279187 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rzskr_7e90d28e-739e-4b37-acdb-6997186a73e8/kube-rbac-proxy/0.log" Jul 27 18:18:58.302142 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:58.302117 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rzskr_7e90d28e-739e-4b37-acdb-6997186a73e8/exporter/0.log" Jul 27 18:18:58.325002 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:18:58.324982 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rzskr_7e90d28e-739e-4b37-acdb-6997186a73e8/extractor/0.log" Jul 27 18:19:00.502891 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:19:00.502857 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-84fc959947-q2kzj_e8295418-67ce-4ee0-87f0-189cd02be6d3/manager/0.log" Jul 27 18:19:00.523537 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:19:00.523498 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-z6jd2_1c56b11d-42a2-4a68-900a-704bf8d4bb8c/server/0.log" Jul 27 18:19:02.053573 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:19:02.053536 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mlzzr/perf-node-gather-daemonset-n6zhz" Jul 27 18:19:06.087871 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:19:06.087842 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4mbmr_b9d10075-b019-4d6d-9904-bdccbc7e8ca1/kube-multus/0.log" Jul 27 18:19:06.264060 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:19:06.264035 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qp76t_c1bf323d-120b-45ff-9117-f8142039ccfc/kube-multus-additional-cni-plugins/0.log" Jul 27 18:19:06.287841 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:19:06.287814 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qp76t_c1bf323d-120b-45ff-9117-f8142039ccfc/egress-router-binary-copy/0.log" Jul 27 18:19:06.313272 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:19:06.313238 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qp76t_c1bf323d-120b-45ff-9117-f8142039ccfc/cni-plugins/0.log" Jul 27 18:19:06.335378 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:19:06.335344 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qp76t_c1bf323d-120b-45ff-9117-f8142039ccfc/bond-cni-plugin/0.log" Jul 27 18:19:06.358472 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:19:06.358444 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qp76t_c1bf323d-120b-45ff-9117-f8142039ccfc/routeoverride-cni/0.log" Jul 27 18:19:06.379266 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:19:06.379244 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qp76t_c1bf323d-120b-45ff-9117-f8142039ccfc/whereabouts-cni-bincopy/0.log" Jul 27 18:19:06.400210 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:19:06.400186 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qp76t_c1bf323d-120b-45ff-9117-f8142039ccfc/whereabouts-cni/0.log" Jul 27 18:19:06.658166 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:19:06.658127 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-69zjf_5115219e-8201-45d1-8a3f-6db85d5e741d/network-metrics-daemon/0.log" Jul 27 18:19:06.678150 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:19:06.678123 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-69zjf_5115219e-8201-45d1-8a3f-6db85d5e741d/kube-rbac-proxy/0.log" Jul 27 18:19:07.677919 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:19:07.677894 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nh6qf_4dbb97f8-c6bc-44e9-b468-e3577bb468af/ovn-controller/0.log" Jul 27 18:19:07.697704 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:19:07.697681 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nh6qf_4dbb97f8-c6bc-44e9-b468-e3577bb468af/ovn-acl-logging/0.log" Jul 27 18:19:07.720660 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:19:07.720640 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nh6qf_4dbb97f8-c6bc-44e9-b468-e3577bb468af/kube-rbac-proxy-node/0.log" Jul 27 18:19:07.743507 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:19:07.743480 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nh6qf_4dbb97f8-c6bc-44e9-b468-e3577bb468af/kube-rbac-proxy-ovn-metrics/0.log" Jul 27 18:19:07.764065 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:19:07.764043 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nh6qf_4dbb97f8-c6bc-44e9-b468-e3577bb468af/northd/0.log" Jul 27 18:19:07.786645 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:19:07.786619 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nh6qf_4dbb97f8-c6bc-44e9-b468-e3577bb468af/nbdb/0.log" Jul 27 18:19:07.807185 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:19:07.807158 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nh6qf_4dbb97f8-c6bc-44e9-b468-e3577bb468af/sbdb/0.log" Jul 27 18:19:07.868627 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:19:07.868591 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nh6qf_4dbb97f8-c6bc-44e9-b468-e3577bb468af/ovnkube-controller/0.log" Jul 27 18:19:09.076918 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:19:09.076888 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-jhzgt_ec0c6ea9-94a4-41b6-9e1f-cccf978294ab/network-check-target-container/0.log" Jul 27 18:19:10.042112 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:19:10.042078 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-8smpx_9b6c93b0-6e89-41e6-accd-cbacae11cf4e/iptables-alerter/0.log" Jul 27 18:19:10.707459 ip-10-0-140-185 kubenswrapper[2570]: I0727 18:19:10.707433 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-w4d45_44c736a9-a8db-4d86-8fb1-bb5287d8c319/tuned/0.log"