Jul 09 09:31:28.195050 ip-10-0-135-204 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 09 09:31:28.195063 ip-10-0-135-204 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 09 09:31:28.195070 ip-10-0-135-204 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 09 09:31:28.195369 ip-10-0-135-204 systemd[1]: Failed to start Kubernetes Kubelet. Jul 09 09:31:38.444458 ip-10-0-135-204 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 09 09:31:38.444479 ip-10-0-135-204 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot cd67205582cc4d498cdafb150a9ce323 -- Jul 09 09:33:33.728728 ip-10-0-135-204 systemd[1]: Starting Kubernetes Kubelet... Jul 09 09:33:34.145380 ip-10-0-135-204 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 09:33:34.145380 ip-10-0-135-204 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 09 09:33:34.145380 ip-10-0-135-204 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 09:33:34.145380 ip-10-0-135-204 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 09 09:33:34.145380 ip-10-0-135-204 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 09:33:34.146305 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.146222 2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 09 09:33:34.151697 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151680 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 09:33:34.151697 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151695 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 09:33:34.151770 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151701 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 09:33:34.151770 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151706 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 09:33:34.151770 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151711 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 09:33:34.151770 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151715 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 09:33:34.151770 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151718 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 09:33:34.151770 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151721 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 09:33:34.151770 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151724 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 09:33:34.151770 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151727 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 09:33:34.151770 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151730 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 09:33:34.151770 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151733 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 09:33:34.151770 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151736 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 09:33:34.151770 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151739 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 09:33:34.151770 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151741 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 09:33:34.151770 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151744 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 09:33:34.151770 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151747 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 09:33:34.151770 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151750 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 09:33:34.151770 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151753 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 09:33:34.151770 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151756 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 09:33:34.151770 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151759 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 09:33:34.152255 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151762 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 09:33:34.152255 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151765 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 09:33:34.152255 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151768 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 09:33:34.152255 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151770 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 09:33:34.152255 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151774 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 09:33:34.152255 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151777 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 09:33:34.152255 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151779 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 09:33:34.152255 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151782 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 09:33:34.152255 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151785 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 09:33:34.152255 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151789 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 09:33:34.152255 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151791 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 09:33:34.152255 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151794 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 09:33:34.152255 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151796 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 09:33:34.152255 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151799 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 09:33:34.152255 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151802 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 09:33:34.152255 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151804 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 09:33:34.152255 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151807 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 09:33:34.152255 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151809 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 09:33:34.152255 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151812 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 09:33:34.152255 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151814 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 09:33:34.152755 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151817 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 09:33:34.152755 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151819 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 09:33:34.152755 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151822 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 09:33:34.152755 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151825 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 09:33:34.152755 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151828 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 09:33:34.152755 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151830 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 09:33:34.152755 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151833 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 09:33:34.152755 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151835 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 09:33:34.152755 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151838 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 09:33:34.152755 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151840 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 09:33:34.152755 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151843 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 09:33:34.152755 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151845 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 09:33:34.152755 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151849 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 09:33:34.152755 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151852 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 09:33:34.152755 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151854 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 09:33:34.152755 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151857 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 09:33:34.152755 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151860 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 09:33:34.152755 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151862 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 09:33:34.152755 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151865 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 09:33:34.152755 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151867 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 09:33:34.153459 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151870 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 09:33:34.153459 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151872 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 09:33:34.153459 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151875 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 09:33:34.153459 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151878 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 09:33:34.153459 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151880 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 09:33:34.153459 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151884 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 09:33:34.153459 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151886 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 09:33:34.153459 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151889 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 09:33:34.153459 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151891 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 09:33:34.153459 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151894 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 09:33:34.153459 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151898 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 09:33:34.153459 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151901 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 09:33:34.153459 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151904 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 09:33:34.153459 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151907 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 09 09:33:34.153459 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151909 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 09:33:34.153459 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151913 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 09:33:34.153459 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151915 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 09:33:34.153459 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151918 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 09:33:34.153459 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151920 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 09:33:34.153459 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151923 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 09:33:34.154109 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151926 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 09:33:34.154109 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151928 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 09:33:34.154109 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151931 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 09:33:34.154109 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151933 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 09:33:34.154109 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151936 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 09:33:34.154109 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151940 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 09:33:34.154109 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151943 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 09:33:34.154109 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151945 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 09:33:34.154109 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151948 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 09:33:34.154109 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151950 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 09:33:34.154109 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151953 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 09:33:34.154109 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151955 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 09:33:34.154109 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151957 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 09:33:34.154109 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151960 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 09:33:34.154109 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151963 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 09:33:34.154109 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151966 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 09:33:34.154109 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151968 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 09:33:34.154109 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.151971 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 09:33:34.154109 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.152638 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 09:33:34.154109 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.152644 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 09:33:34.154626 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.152647 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 09:33:34.154626 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.152650 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 09:33:34.154626 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.152653 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 09:33:34.154626 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.152656 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 09:33:34.154626 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.152658 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 09:33:34.154626 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.152661 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 09:33:34.154626 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.152664 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 09:33:34.154626 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.152667 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 09:33:34.154626 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.152669 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 09:33:34.154626 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.152672 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 09:33:34.154626 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.152674 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 09:33:34.154626 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.152677 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 09:33:34.154626 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.152680 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 09:33:34.154626 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.152683 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 09:33:34.154626 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.152685 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 09:33:34.154626 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.152688 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 09:33:34.154626 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.152690 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 09:33:34.154626 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.152693 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 09:33:34.154626 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.152696 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 09:33:34.155133 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.152698 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 09:33:34.155133 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.152701 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 09:33:34.155133 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.152703 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 09:33:34.155133 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.152706 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 09:33:34.155133 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.152709 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 09:33:34.155133 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.152712 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 09:33:34.155133 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.152714 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 09:33:34.155133 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.152719 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 09:33:34.155133 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.152722 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 09:33:34.155133 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.152764 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 09:33:34.155133 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.152804 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 09:33:34.155133 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.152808 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 09:33:34.155133 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.152811 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 09:33:34.155133 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.152815 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 09:33:34.155133 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.152818 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 09:33:34.155133 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.152822 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 09:33:34.155133 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.152824 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 09:33:34.155133 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.152827 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 09:33:34.155133 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.152830 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 09:33:34.155133 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.152833 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 09:33:34.155657 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.152836 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 09:33:34.155657 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.152839 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 09:33:34.155657 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.152842 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 09:33:34.155657 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.152848 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 09:33:34.155657 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.152851 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 09:33:34.155657 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.152853 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 09:33:34.155657 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.152856 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 09:33:34.155657 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.152859 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 09:33:34.155657 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.152862 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 09:33:34.155657 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.152865 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 09:33:34.155657 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.152869 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 09:33:34.155657 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.153825 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 09:33:34.155657 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.153838 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 09:33:34.155657 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.153844 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 09:33:34.155657 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.153855 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 09:33:34.155657 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.153862 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 09:33:34.155657 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.153869 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 09:33:34.155657 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.153874 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 09:33:34.155657 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.153879 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 09:33:34.156116 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.153884 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 09:33:34.156116 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.153888 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 09:33:34.156116 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.153903 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 09:33:34.156116 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.153908 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 09 09:33:34.156116 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.153913 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 09:33:34.156116 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.153918 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 09:33:34.156116 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.153923 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 09:33:34.156116 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.153931 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 09:33:34.156116 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.153934 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 09:33:34.156116 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.153937 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 09:33:34.156116 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.153940 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 09:33:34.156116 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.153943 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 09:33:34.156116 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.153946 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 09:33:34.156116 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.153952 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 09:33:34.156116 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.153955 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 09:33:34.156116 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.153957 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 09:33:34.156116 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.153960 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 09:33:34.156116 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.153980 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 09:33:34.156116 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.153983 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 09:33:34.156116 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154065 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 09:33:34.156622 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154069 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 09:33:34.156622 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154072 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 09:33:34.156622 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154075 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 09:33:34.156622 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154078 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 09:33:34.156622 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154083 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 09:33:34.156622 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154086 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 09:33:34.156622 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154089 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 09:33:34.156622 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154091 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 09:33:34.156622 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154094 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 09:33:34.156622 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154097 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 09:33:34.156622 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154101 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 09:33:34.156622 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154104 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 09:33:34.156622 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154106 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 09:33:34.156622 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154109 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 09:33:34.156622 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154112 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 09:33:34.156622 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154114 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 09:33:34.156622 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154117 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 09:33:34.156622 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154120 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 09:33:34.156622 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154127 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 09:33:34.156622 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154555 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 09:33:34.157105 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154561 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 09:33:34.157105 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154565 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 09:33:34.157105 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154568 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 09:33:34.157105 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154571 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 09:33:34.157105 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154574 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 09:33:34.157105 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154577 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 09 09:33:34.157105 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154580 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 09:33:34.157105 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154582 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 09:33:34.157105 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154585 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 09:33:34.157105 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154587 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 09:33:34.157105 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154591 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 09:33:34.157105 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154593 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 09:33:34.157105 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154596 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 09:33:34.157105 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154598 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 09:33:34.157105 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154601 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 09:33:34.157105 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154604 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 09:33:34.157105 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154607 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 09:33:34.157105 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154609 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 09:33:34.157105 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154612 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 09:33:34.157105 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154615 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 09:33:34.157601 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154618 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 09:33:34.157601 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154620 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 09:33:34.157601 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154623 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 09:33:34.157601 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154626 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 09:33:34.157601 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154628 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 09:33:34.157601 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154631 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 09:33:34.157601 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154634 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 09:33:34.157601 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154637 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 09:33:34.157601 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154639 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 09:33:34.157601 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154642 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 09:33:34.157601 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154645 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 09:33:34.157601 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154648 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 09:33:34.157601 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154651 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 09:33:34.157601 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154653 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 09:33:34.157601 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154656 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 09:33:34.157601 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154658 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 09:33:34.157601 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154661 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 09:33:34.157601 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154664 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 09:33:34.157601 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154666 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 09:33:34.158063 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154669 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 09:33:34.158063 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154671 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 09:33:34.158063 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154674 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 09:33:34.158063 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154676 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 09:33:34.158063 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154679 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 09:33:34.158063 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154681 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 09:33:34.158063 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154684 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 09:33:34.158063 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154687 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 09:33:34.158063 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154690 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 09:33:34.158063 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154693 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 09:33:34.158063 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154696 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 09:33:34.158063 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154699 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 09:33:34.158063 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154701 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 09:33:34.158063 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154703 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 09:33:34.158063 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154706 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 09:33:34.158063 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154708 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 09:33:34.158063 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154711 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 09:33:34.158063 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154715 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 09:33:34.158063 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154719 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 09:33:34.158556 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154722 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 09:33:34.158556 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154725 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 09:33:34.158556 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154727 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 09:33:34.158556 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154731 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 09:33:34.158556 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154735 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 09:33:34.158556 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154738 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 09:33:34.158556 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154741 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 09:33:34.158556 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154745 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 09:33:34.158556 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154749 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 09:33:34.158556 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154752 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 09:33:34.158556 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154755 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 09:33:34.158556 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154758 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 09:33:34.158556 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154761 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 09:33:34.158556 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154764 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 09:33:34.158556 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154766 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 09:33:34.158556 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154769 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 09:33:34.158556 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154772 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 09:33:34.158556 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154774 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 09:33:34.158556 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154776 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 09:33:34.159021 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154779 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 09:33:34.159021 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154782 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 09:33:34.159021 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154786 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 09:33:34.159021 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154788 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 09:33:34.159021 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154791 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 09:33:34.159021 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154793 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 09:33:34.159021 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154796 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 09:33:34.159021 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154798 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 09:33:34.159021 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154801 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 09:33:34.159021 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154804 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 09:33:34.159021 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154807 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 09:33:34.159021 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154809 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 09:33:34.159021 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154812 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 09:33:34.159021 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154814 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 09:33:34.159021 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154817 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 09:33:34.159021 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154820 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 09:33:34.159021 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154822 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 09:33:34.159021 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154825 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 09:33:34.159021 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154827 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 09:33:34.159021 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154830 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 09:33:34.159532 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.154832 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 09:33:34.159532 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.154899 2578 flags.go:64] FLAG: --address="0.0.0.0" Jul 09 09:33:34.159532 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.154905 2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 09 09:33:34.159532 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.154910 2578 flags.go:64] FLAG: --anonymous-auth="true" Jul 09 09:33:34.159532 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.154914 2578 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 09 09:33:34.159532 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.154918 2578 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 09 09:33:34.159532 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.154921 2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 09 09:33:34.159532 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.154925 2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 09 09:33:34.159532 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.154929 2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 09 09:33:34.159532 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.154932 2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 09 09:33:34.159532 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.154935 2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 09 09:33:34.159532 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.154938 2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 09 09:33:34.159532 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.154942 2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 09 09:33:34.159532 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.154945 2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 09 09:33:34.159532 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.154948 2578 flags.go:64] FLAG: --cgroup-root="" Jul 09 09:33:34.159532 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.154951 2578 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 09 09:33:34.159532 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.154954 2578 flags.go:64] FLAG: --client-ca-file="" Jul 09 09:33:34.159532 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.154956 2578 flags.go:64] FLAG: --cloud-provider="external" Jul 09 09:33:34.159532 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.154959 2578 flags.go:64] FLAG: --cluster-dns="[]" Jul 09 09:33:34.159532 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.154964 2578 flags.go:64] FLAG: --cluster-domain="" Jul 09 09:33:34.159532 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.154967 2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 09 09:33:34.159532 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.154970 2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 09 09:33:34.159532 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.154973 2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 09 09:33:34.159532 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.154976 2578 flags.go:64] FLAG: --container-log-max-files="5" Jul 09 09:33:34.160113 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.154980 2578 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 09 09:33:34.160113 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.154983 2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 09 09:33:34.160113 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.154985 2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 09 09:33:34.160113 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.154989 2578 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 09 09:33:34.160113 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.154992 2578 flags.go:64] FLAG: --contention-profiling="false" Jul 09 09:33:34.160113 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.154994 2578 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 09 09:33:34.160113 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.154999 2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 09 09:33:34.160113 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155002 2578 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 09 09:33:34.160113 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155005 2578 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 09 09:33:34.160113 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155009 2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 09 09:33:34.160113 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155012 2578 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 09 09:33:34.160113 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155015 2578 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 09 09:33:34.160113 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155017 2578 flags.go:64] FLAG: --enable-load-reader="false" Jul 09 09:33:34.160113 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155020 2578 flags.go:64] FLAG: --enable-server="true" Jul 09 09:33:34.160113 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155023 2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 09 09:33:34.160113 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155028 2578 flags.go:64] FLAG: --event-burst="100" Jul 09 09:33:34.160113 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155030 2578 flags.go:64] FLAG: --event-qps="50" Jul 09 09:33:34.160113 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155033 2578 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 09 09:33:34.160113 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155036 2578 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 09 09:33:34.160113 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155039 2578 flags.go:64] FLAG: --eviction-hard="" Jul 09 09:33:34.160113 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155043 2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 09 09:33:34.160113 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155045 2578 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 09 09:33:34.160113 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155048 2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 09 09:33:34.160113 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155052 2578 flags.go:64] FLAG: --eviction-soft="" Jul 09 09:33:34.160113 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155055 2578 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 09 09:33:34.160765 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155058 2578 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 09 09:33:34.160765 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155061 2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 09 09:33:34.160765 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155064 2578 flags.go:64] FLAG: --experimental-mounter-path="" Jul 09 09:33:34.160765 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155067 2578 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 09 09:33:34.160765 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155069 2578 flags.go:64] FLAG: --fail-swap-on="true" Jul 09 09:33:34.160765 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155072 2578 flags.go:64] FLAG: --feature-gates="" Jul 09 09:33:34.160765 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155076 2578 flags.go:64] FLAG: --file-check-frequency="20s" Jul 09 09:33:34.160765 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155079 2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 09 09:33:34.160765 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155082 2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 09 09:33:34.160765 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155085 2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 09 09:33:34.160765 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155088 2578 flags.go:64] FLAG: --healthz-port="10248" Jul 09 09:33:34.160765 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155091 2578 flags.go:64] FLAG: --help="false" Jul 09 09:33:34.160765 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155093 2578 flags.go:64] FLAG: --hostname-override="ip-10-0-135-204.ec2.internal" Jul 09 09:33:34.160765 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155098 2578 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 09 09:33:34.160765 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155101 2578 flags.go:64] FLAG: --http-check-frequency="20s" Jul 09 09:33:34.160765 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155104 2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 09 09:33:34.160765 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155108 2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 09 09:33:34.160765 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155111 2578 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 09 09:33:34.160765 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155114 2578 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 09 09:33:34.160765 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155116 2578 flags.go:64] FLAG: --image-service-endpoint="" Jul 09 09:33:34.160765 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155119 2578 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 09 09:33:34.160765 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155122 2578 flags.go:64] FLAG: --kube-api-burst="100" Jul 09 09:33:34.160765 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155124 2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 09 09:33:34.161350 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155127 2578 flags.go:64] FLAG: --kube-api-qps="50" Jul 09 09:33:34.161350 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155130 2578 flags.go:64] FLAG: --kube-reserved="" Jul 09 09:33:34.161350 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155133 2578 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 09 09:33:34.161350 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155136 2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 09 09:33:34.161350 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155139 2578 flags.go:64] FLAG: --kubelet-cgroups="" Jul 09 09:33:34.161350 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155142 2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 09 09:33:34.161350 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155145 2578 flags.go:64] FLAG: --lock-file="" Jul 09 09:33:34.161350 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155148 2578 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 09 09:33:34.161350 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155151 2578 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 09 09:33:34.161350 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155154 2578 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 09 09:33:34.161350 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155163 2578 flags.go:64] FLAG: --log-json-split-stream="false" Jul 09 09:33:34.161350 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155166 2578 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 09 09:33:34.161350 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155169 2578 flags.go:64] FLAG: --log-text-split-stream="false" Jul 09 09:33:34.161350 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155172 2578 flags.go:64] FLAG: --logging-format="text" Jul 09 09:33:34.161350 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155174 2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 09 09:33:34.161350 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155178 2578 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 09 09:33:34.161350 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155180 2578 flags.go:64] FLAG: --manifest-url="" Jul 09 09:33:34.161350 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155183 2578 flags.go:64] FLAG: --manifest-url-header="" Jul 09 09:33:34.161350 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155187 2578 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 09 09:33:34.161350 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155190 2578 flags.go:64] FLAG: --max-open-files="1000000" Jul 09 09:33:34.161350 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155194 2578 flags.go:64] FLAG: --max-pods="110" Jul 09 09:33:34.161350 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155196 2578 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 09 09:33:34.161350 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155201 2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 09 09:33:34.161350 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155204 2578 flags.go:64] FLAG: --memory-manager-policy="None" Jul 09 09:33:34.161350 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155206 2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 09 09:33:34.161965 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155209 2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 09 09:33:34.161965 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155212 2578 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 09 09:33:34.161965 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155215 2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 09 09:33:34.161965 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155222 2578 flags.go:64] FLAG: --node-status-max-images="50" Jul 09 09:33:34.161965 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155225 2578 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 09 09:33:34.161965 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155228 2578 flags.go:64] FLAG: --oom-score-adj="-999" Jul 09 09:33:34.161965 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155231 2578 flags.go:64] FLAG: --pod-cidr="" Jul 09 09:33:34.161965 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155233 2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 09 09:33:34.161965 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155239 2578 flags.go:64] FLAG: --pod-manifest-path="" Jul 09 09:33:34.161965 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155242 2578 flags.go:64] FLAG: --pod-max-pids="-1" Jul 09 09:33:34.161965 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155245 2578 flags.go:64] FLAG: --pods-per-core="0" Jul 09 09:33:34.161965 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155248 2578 flags.go:64] FLAG: --port="10250" Jul 09 09:33:34.161965 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155251 2578 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 09 09:33:34.161965 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155254 2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03d53313b31dfb3fe" Jul 09 09:33:34.161965 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155257 2578 flags.go:64] FLAG: --qos-reserved="" Jul 09 09:33:34.161965 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155260 2578 flags.go:64] FLAG: --read-only-port="10255" Jul 09 09:33:34.161965 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155263 2578 flags.go:64] FLAG: --register-node="true" Jul 09 09:33:34.161965 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155266 2578 flags.go:64] FLAG: --register-with-taints="" Jul 09 09:33:34.161965 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155269 2578 flags.go:64] FLAG: --registry-burst="10" Jul 09 09:33:34.161965 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155272 2578 flags.go:64] FLAG: --registry-qps="5" Jul 09 09:33:34.161965 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155275 2578 flags.go:64] FLAG: --reserved-cpus="" Jul 09 09:33:34.161965 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155278 2578 flags.go:64] FLAG: --reserved-memory="" Jul 09 09:33:34.161965 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155281 2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 09 09:33:34.161965 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155284 2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 09 09:33:34.161965 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155287 2578 flags.go:64] FLAG: --rotate-certificates="false" Jul 09 09:33:34.162588 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155290 2578 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 09 09:33:34.162588 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155292 2578 flags.go:64] FLAG: --runonce="false" Jul 09 09:33:34.162588 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155295 2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 09 09:33:34.162588 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155298 2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 09 09:33:34.162588 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155303 2578 flags.go:64] FLAG: --seccomp-default="false" Jul 09 09:33:34.162588 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155305 2578 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 09 09:33:34.162588 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155308 2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 09 09:33:34.162588 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155312 2578 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 09 09:33:34.162588 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155314 2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 09 09:33:34.162588 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155317 2578 flags.go:64] FLAG: --storage-driver-password="root" Jul 09 09:33:34.162588 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155320 2578 flags.go:64] FLAG: --storage-driver-secure="false" Jul 09 09:33:34.162588 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155323 2578 flags.go:64] FLAG: --storage-driver-table="stats" Jul 09 09:33:34.162588 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155326 2578 flags.go:64] FLAG: --storage-driver-user="root" Jul 09 09:33:34.162588 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155329 2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 09 09:33:34.162588 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155332 2578 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 09 09:33:34.162588 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155335 2578 flags.go:64] FLAG: --system-cgroups="" Jul 09 09:33:34.162588 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155337 2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 09 09:33:34.162588 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155342 2578 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 09 09:33:34.162588 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155358 2578 flags.go:64] FLAG: --tls-cert-file="" Jul 09 09:33:34.162588 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155361 2578 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 09 09:33:34.162588 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155366 2578 flags.go:64] FLAG: --tls-min-version="" Jul 09 09:33:34.162588 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155369 2578 flags.go:64] FLAG: --tls-private-key-file="" Jul 09 09:33:34.162588 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155371 2578 flags.go:64] FLAG: --topology-manager-policy="none" Jul 09 09:33:34.162588 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155374 2578 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 09 09:33:34.162588 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155377 2578 flags.go:64] FLAG: --topology-manager-scope="container" Jul 09 09:33:34.163191 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155380 2578 flags.go:64] FLAG: --v="2" Jul 09 09:33:34.163191 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155384 2578 flags.go:64] FLAG: --version="false" Jul 09 09:33:34.163191 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155388 2578 flags.go:64] FLAG: --vmodule="" Jul 09 09:33:34.163191 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155391 2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 09 09:33:34.163191 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.155395 2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 09 09:33:34.163191 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155529 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 09:33:34.163191 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155533 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 09:33:34.163191 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155536 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 09:33:34.163191 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155539 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 09:33:34.163191 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155541 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 09:33:34.163191 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155544 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 09:33:34.163191 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155549 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 09:33:34.163191 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155551 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 09:33:34.163191 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155554 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 09:33:34.163191 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155556 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 09:33:34.163191 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155559 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 09:33:34.163191 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155562 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 09:33:34.163191 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155565 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 09:33:34.163191 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155567 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 09:33:34.163191 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155571 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 09:33:34.163191 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155575 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 09:33:34.163772 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155578 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 09 09:33:34.163772 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155581 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 09:33:34.163772 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155584 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 09:33:34.163772 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155587 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 09:33:34.163772 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155591 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 09:33:34.163772 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155593 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 09:33:34.163772 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155596 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 09:33:34.163772 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155599 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 09:33:34.163772 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155601 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 09:33:34.163772 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155603 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 09:33:34.163772 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155606 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 09:33:34.163772 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155609 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 09:33:34.163772 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155612 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 09:33:34.163772 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155615 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 09:33:34.163772 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155618 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 09:33:34.163772 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155620 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 09:33:34.163772 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155623 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 09:33:34.163772 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155627 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 09:33:34.163772 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155631 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 09:33:34.164233 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155633 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 09:33:34.164233 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155636 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 09:33:34.164233 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155639 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 09:33:34.164233 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155642 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 09:33:34.164233 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155645 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 09:33:34.164233 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155647 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 09:33:34.164233 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155650 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 09:33:34.164233 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155652 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 09:33:34.164233 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155655 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 09:33:34.164233 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155657 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 09:33:34.164233 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155660 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 09:33:34.164233 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155663 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 09:33:34.164233 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155667 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 09:33:34.164233 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155670 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 09:33:34.164233 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155673 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 09:33:34.164233 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155676 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 09:33:34.164233 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155679 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 09:33:34.164233 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155683 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 09:33:34.164233 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155686 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 09:33:34.164713 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155689 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 09:33:34.164713 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155692 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 09:33:34.164713 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155694 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 09:33:34.164713 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155697 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 09:33:34.164713 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155700 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 09:33:34.164713 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155702 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 09:33:34.164713 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155705 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 09:33:34.164713 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155708 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 09:33:34.164713 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155710 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 09:33:34.164713 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155713 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 09:33:34.164713 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155715 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 09:33:34.164713 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155718 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 09:33:34.164713 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155720 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 09:33:34.164713 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155723 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 09:33:34.164713 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155725 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 09:33:34.164713 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155728 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 09:33:34.164713 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155732 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 09:33:34.164713 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155734 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 09:33:34.164713 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155737 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 09:33:34.164713 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155739 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 09:33:34.164713 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155742 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 09:33:34.165217 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155745 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 09:33:34.165217 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155747 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 09:33:34.165217 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155750 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 09:33:34.165217 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155752 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 09:33:34.165217 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155755 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 09:33:34.165217 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155757 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 09:33:34.165217 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155760 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 09:33:34.165217 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155763 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 09:33:34.165217 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155765 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 09:33:34.165217 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155770 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 09:33:34.165217 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155773 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 09:33:34.165217 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155775 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 09:33:34.165217 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155778 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 09:33:34.165217 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155780 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 09:33:34.165217 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155783 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 09:33:34.165217 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155786 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 09:33:34.165217 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155789 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 09:33:34.165217 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155791 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 09:33:34.165217 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155794 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 09:33:34.165217 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155797 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 09:33:34.165772 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155799 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 09:33:34.165772 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155802 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 09:33:34.165772 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155804 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 09:33:34.165772 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.155807 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 09:33:34.165772 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.156394 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 09:33:34.165772 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.163143 2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 09 09:33:34.165772 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.163157 2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 09 09:33:34.165772 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.164682 2578 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 09 09:33:34.165772 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.164691 2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 09 09:33:34.165772 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164746 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 09:33:34.165772 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164751 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 09:33:34.165772 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164754 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 09:33:34.165772 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164758 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 09:33:34.165772 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164761 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 09:33:34.165772 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164764 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 09:33:34.166142 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164767 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 09:33:34.166142 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164770 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 09:33:34.166142 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164773 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 09:33:34.166142 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164775 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 09:33:34.166142 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164778 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 09:33:34.166142 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164781 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 09:33:34.166142 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164783 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 09:33:34.166142 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164786 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 09:33:34.166142 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164790 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 09:33:34.166142 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164794 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 09:33:34.166142 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164798 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 09:33:34.166142 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164802 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 09:33:34.166142 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164805 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 09:33:34.166142 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164808 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 09:33:34.166142 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164810 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 09:33:34.166142 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164814 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 09:33:34.166142 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164818 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 09:33:34.166142 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164821 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 09:33:34.166142 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164823 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 09:33:34.166605 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164826 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 09:33:34.166605 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164829 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 09:33:34.166605 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164831 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 09:33:34.166605 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164834 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 09:33:34.166605 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164837 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 09:33:34.166605 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164840 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 09:33:34.166605 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164843 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 09:33:34.166605 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164846 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 09:33:34.166605 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164848 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 09:33:34.166605 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164851 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 09:33:34.166605 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164854 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 09:33:34.166605 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164856 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 09:33:34.166605 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164859 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 09:33:34.166605 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164861 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 09:33:34.166605 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164864 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 09:33:34.166605 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164867 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 09:33:34.166605 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164870 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 09:33:34.166605 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164872 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 09:33:34.166605 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164874 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 09:33:34.166605 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164877 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 09:33:34.167081 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164879 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 09:33:34.167081 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164882 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 09:33:34.167081 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164885 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 09:33:34.167081 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164887 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 09:33:34.167081 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164890 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 09:33:34.167081 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164892 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 09:33:34.167081 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164894 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 09:33:34.167081 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164897 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 09:33:34.167081 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164899 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 09:33:34.167081 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164903 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 09:33:34.167081 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164905 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 09:33:34.167081 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164907 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 09:33:34.167081 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164910 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 09:33:34.167081 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164912 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 09:33:34.167081 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164915 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 09 09:33:34.167081 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164917 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 09:33:34.167081 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164920 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 09:33:34.167081 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164925 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 09:33:34.167081 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164929 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 09:33:34.167081 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164931 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 09:33:34.167579 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164934 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 09:33:34.167579 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164937 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 09:33:34.167579 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164939 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 09:33:34.167579 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164941 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 09:33:34.167579 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164944 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 09:33:34.167579 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164947 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 09:33:34.167579 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164949 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 09:33:34.167579 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164952 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 09:33:34.167579 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164954 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 09:33:34.167579 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164957 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 09:33:34.167579 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164959 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 09:33:34.167579 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164962 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 09:33:34.167579 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164964 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 09:33:34.167579 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164967 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 09:33:34.167579 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164970 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 09:33:34.167579 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164972 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 09:33:34.167579 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164974 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 09:33:34.167579 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164977 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 09:33:34.167579 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164979 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 09:33:34.167579 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164982 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 09:33:34.168054 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164984 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 09:33:34.168054 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164986 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 09:33:34.168054 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164989 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 09:33:34.168054 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164992 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 09:33:34.168054 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164994 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 09:33:34.168054 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164997 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 09:33:34.168054 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.164999 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 09:33:34.168054 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165002 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 09:33:34.168054 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165005 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 09:33:34.168054 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165007 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 09:33:34.168054 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165010 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 09:33:34.168054 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165012 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 09:33:34.168054 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165014 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 09:33:34.168054 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165017 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 09:33:34.168054 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.165022 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 09:33:34.168422 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165522 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 09:33:34.168422 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165528 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 09:33:34.168422 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165531 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 09:33:34.168422 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165534 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 09:33:34.168422 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165537 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 09:33:34.168422 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165540 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 09:33:34.168422 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165543 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 09 09:33:34.168422 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165545 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 09:33:34.168422 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165548 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 09:33:34.168422 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165550 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 09:33:34.168422 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165553 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 09:33:34.168422 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165555 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 09:33:34.168422 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165557 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 09:33:34.168422 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165560 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 09:33:34.168422 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165562 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 09:33:34.168422 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165565 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 09:33:34.168422 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165567 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 09:33:34.168422 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165570 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 09:33:34.168422 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165572 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 09:33:34.168422 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165574 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 09:33:34.168906 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165577 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 09:33:34.168906 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165580 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 09:33:34.168906 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165582 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 09:33:34.168906 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165585 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 09:33:34.168906 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165588 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 09:33:34.168906 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165590 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 09:33:34.168906 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165593 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 09:33:34.168906 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165595 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 09:33:34.168906 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165598 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 09:33:34.168906 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165600 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 09:33:34.168906 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165603 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 09:33:34.168906 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165606 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 09:33:34.168906 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165608 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 09:33:34.168906 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165610 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 09:33:34.168906 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165613 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 09:33:34.168906 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165615 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 09:33:34.168906 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165618 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 09:33:34.168906 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165620 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 09:33:34.168906 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165623 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 09:33:34.169403 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165625 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 09:33:34.169403 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165627 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 09:33:34.169403 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165630 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 09:33:34.169403 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165632 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 09:33:34.169403 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165635 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 09:33:34.169403 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165639 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 09:33:34.169403 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165642 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 09:33:34.169403 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165645 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 09:33:34.169403 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165648 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 09:33:34.169403 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165651 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 09:33:34.169403 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165654 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 09:33:34.169403 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165657 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 09:33:34.169403 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165659 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 09:33:34.169403 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165662 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 09:33:34.169403 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165665 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 09:33:34.169403 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165668 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 09:33:34.169403 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165671 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 09:33:34.169403 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165673 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 09:33:34.169403 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165677 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 09:33:34.169951 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165679 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 09:33:34.169951 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165682 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 09:33:34.169951 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165684 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 09:33:34.169951 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165687 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 09:33:34.169951 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165689 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 09:33:34.169951 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165692 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 09:33:34.169951 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165694 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 09:33:34.169951 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165698 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 09:33:34.169951 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165701 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 09:33:34.169951 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165704 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 09:33:34.169951 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165707 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 09:33:34.169951 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165711 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 09:33:34.169951 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165714 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 09:33:34.169951 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165717 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 09:33:34.169951 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165719 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 09:33:34.169951 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165722 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 09:33:34.169951 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165725 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 09:33:34.169951 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165727 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 09:33:34.169951 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165729 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 09:33:34.169951 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165732 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 09:33:34.170417 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165734 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 09:33:34.170417 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165737 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 09:33:34.170417 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165739 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 09:33:34.170417 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165742 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 09:33:34.170417 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165744 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 09:33:34.170417 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165747 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 09:33:34.170417 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165749 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 09:33:34.170417 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165751 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 09:33:34.170417 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165754 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 09:33:34.170417 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165757 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 09:33:34.170417 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165759 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 09:33:34.170417 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165762 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 09:33:34.170417 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165764 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 09:33:34.170417 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165767 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 09:33:34.170417 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165770 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 09:33:34.170417 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165772 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 09:33:34.170417 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165775 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 09:33:34.170417 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165778 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 09:33:34.170417 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165780 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 09:33:34.170417 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165783 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 09:33:34.170903 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:34.165785 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 09:33:34.170903 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.165790 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 09:33:34.170903 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.165890 2578 server.go:962] "Client rotation is on, will bootstrap in background" Jul 09 09:33:34.170903 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.168713 2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 09 09:33:34.170903 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.170451 2578 server.go:1019] "Starting client certificate rotation" Jul 09 09:33:34.170903 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.170554 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 09 09:33:34.170903 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.170611 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 09 09:33:34.173973 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.173960 2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 09 09:33:34.175043 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.175024 2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 09 09:33:34.189536 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.189509 2578 log.go:25] "Validated CRI v1 runtime API" Jul 09 09:33:34.191566 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.191549 2578 log.go:25] "Validated CRI v1 image API" Jul 09 09:33:34.192274 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.192259 2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 09 09:33:34.195367 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.195334 2578 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7be549c0-c25c-493f-909d-944b176a033d:/dev/nvme0n1p3 972e7279-6144-4455-9ade-d049e9a0b9ae:/dev/nvme0n1p4] Jul 09 09:33:34.195424 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.195368 2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 09 09:33:34.200544 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.200526 2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 09 09:33:34.200721 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.200611 2578 manager.go:216] Machine: {Timestamp:2026-07-09 09:33:34.199409053 +0000 UTC m=+0.350564187 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100309 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c672eecd7032eacd17c30a72a407b SystemUUID:ec2c672e-ecd7-032e-acd1-7c30a72a407b BootID:cd672055-82cc-4d49-8cda-fb150a9ce323 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:5f:e1:68:da:15 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:5f:e1:68:da:15 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5a:2d:08:6e:f0:2b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 09 09:33:34.200721 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.200718 2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 09 09:33:34.200817 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.200794 2578 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 09 09:33:34.206091 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.206069 2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 09 09:33:34.206229 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.206093 2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-204.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 09 09:33:34.207423 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.207413 2578 topology_manager.go:138] "Creating topology manager with none policy" Jul 09 09:33:34.207470 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.207426 2578 container_manager_linux.go:309] "Creating device plugin manager" Jul 09 09:33:34.207470 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.207434 2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 09 09:33:34.208057 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.208047 2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 09 09:33:34.209387 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.209378 2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 09 09:33:34.212641 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.212630 2578 state_mem.go:36] "Initialized new in-memory state store" Jul 09 09:33:34.212745 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.212736 2578 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 09 09:33:34.212775 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.212765 2578 kubelet.go:486] "Attempting to sync node with API server" Jul 09 09:33:34.212801 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.212782 2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 09 09:33:34.212801 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.212796 2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 09 09:33:34.212856 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.212810 2578 kubelet.go:398] "Adding apiserver pod source" Jul 09 09:33:34.212856 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.212823 2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 09 09:33:34.213151 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.213140 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 09 09:33:34.213188 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.213158 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 09 09:33:34.215276 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.215255 2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 09 09:33:34.217095 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.217082 2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 09 09:33:34.217135 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.217112 2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 09 09:33:34.218332 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.218317 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 09 09:33:34.218332 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.218332 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 09 09:33:34.218456 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.218339 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 09 09:33:34.218456 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.218344 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 09 09:33:34.218456 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.218361 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 09 09:33:34.218456 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.218367 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 09 09:33:34.218456 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.218375 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 09 09:33:34.218456 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.218379 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 09 09:33:34.218456 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.218386 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 09 09:33:34.218456 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.218391 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 09 09:33:34.218456 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.218405 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 09 09:33:34.218456 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.218410 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 09 09:33:34.219314 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.219303 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 09 09:33:34.219314 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.219313 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 09 09:33:34.222616 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.222602 2578 server.go:1268] "Started kubelet" Jul 09 09:33:34.222696 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.222665 2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 09 09:33:34.222799 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.222748 2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 09 09:33:34.222887 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.222837 2578 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 09 09:33:34.223501 ip-10-0-135-204 systemd[1]: Started Kubernetes Kubelet. Jul 09 09:33:34.223648 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.223573 2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-204.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 09 09:33:34.224247 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.224194 2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 09 09:33:34.224709 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:34.224683 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-204.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 09 09:33:34.224709 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:34.224683 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 09 09:33:34.225013 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.224998 2578 server.go:310] "Adding debug handlers to kubelet server" Jul 09 09:33:34.228604 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:34.228585 2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 09 09:33:34.229986 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.229972 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-shl28" Jul 09 09:33:34.232558 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:34.231556 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-204.ec2.internal.18c0954de4cffa1b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-204.ec2.internal,UID:ip-10-0-135-204.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-204.ec2.internal,},FirstTimestamp:2026-07-09 09:33:34.222572059 +0000 UTC m=+0.373727188,LastTimestamp:2026-07-09 09:33:34.222572059 +0000 UTC m=+0.373727188,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-204.ec2.internal,}" Jul 09 09:33:34.232763 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.232747 2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 09 09:33:34.232821 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.232759 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 09 09:33:34.232872 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:34.232835 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-204.ec2.internal\" not found" Jul 09 09:33:34.233492 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.233470 2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 09 09:33:34.233579 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.233470 2578 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 09 09:33:34.233579 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.233534 2578 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 09 09:33:34.233666 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.233638 2578 factory.go:55] Registering systemd factory Jul 09 09:33:34.233666 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.233651 2578 reconstruct.go:97] "Volume reconstruction finished" Jul 09 09:33:34.233666 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.233659 2578 reconciler.go:29] "Reconciler: start to sync state" Jul 09 09:33:34.233805 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.233678 2578 factory.go:223] Registration of the systemd container factory successfully Jul 09 09:33:34.233875 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.233860 2578 factory.go:153] Registering CRI-O factory Jul 09 09:33:34.233875 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.233872 2578 factory.go:223] Registration of the crio container factory successfully Jul 09 09:33:34.234001 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.233923 2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 09 09:33:34.234001 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.233948 2578 factory.go:103] Registering Raw factory Jul 09 09:33:34.234001 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.233964 2578 manager.go:1196] Started watching for new ooms in manager Jul 09 09:33:34.234600 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.234584 2578 manager.go:349] Starting recovery of all containers Jul 09 09:33:34.237848 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.237823 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-shl28" Jul 09 09:33:34.238821 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:34.238798 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 09 09:33:34.242710 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:34.242684 2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-204.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 09 09:33:34.245430 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.245291 2578 manager.go:354] Recovery completed Jul 09 09:33:34.246885 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:34.246869 2578 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Jul 09 09:33:34.249672 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.249659 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 09:33:34.251452 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.251417 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-204.ec2.internal" event="NodeHasSufficientMemory" Jul 09 09:33:34.251513 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.251467 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-204.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 09:33:34.251513 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.251485 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-204.ec2.internal" event="NodeHasSufficientPID" Jul 09 09:33:34.251900 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.251887 2578 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 09 09:33:34.251900 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.251898 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 09:33:34.252002 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.251901 2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 09 09:33:34.252002 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.251918 2578 state_mem.go:36] "Initialized new in-memory state store" Jul 09 09:33:34.252578 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.252561 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-204.ec2.internal" event="NodeHasSufficientMemory" Jul 09 09:33:34.252660 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.252590 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-204.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 09:33:34.252660 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.252605 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-204.ec2.internal" event="NodeHasSufficientPID" Jul 09 09:33:34.253515 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.253500 2578 policy_none.go:49] "None policy: Start" Jul 09 09:33:34.253515 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.253517 2578 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 09 09:33:34.253605 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.253527 2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 09 09:33:34.254635 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.254625 2578 policy_none.go:47] "Start" Jul 09 09:33:34.287993 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.287973 2578 manager.go:337] "Starting Device Plugin manager" Jul 09 09:33:34.288064 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:34.288002 2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 09 09:33:34.288064 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.288012 2578 server.go:87] "Starting device plugin registration server" Jul 09 09:33:34.288248 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.288234 2578 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 09 09:33:34.288310 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.288249 2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 09 09:33:34.288364 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.288324 2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 09 09:33:34.288471 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.288433 2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 09 09:33:34.288471 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.288463 2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 09 09:33:34.288886 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:34.288867 2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 09 09:33:34.288969 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:34.288911 2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-204.ec2.internal\" not found" Jul 09 09:33:34.375975 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.375947 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 09 09:33:34.377429 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.377414 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 09 09:33:34.377510 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.377467 2578 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 09 09:33:34.377510 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.377488 2578 kubelet.go:2448] "Starting kubelet main sync loop" Jul 09 09:33:34.377573 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:34.377522 2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 09 09:33:34.380153 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.380138 2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 09:33:34.389193 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.389179 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 09:33:34.389979 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.389963 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-204.ec2.internal" event="NodeHasSufficientMemory" Jul 09 09:33:34.390062 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.389994 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-204.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 09:33:34.390062 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.390005 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-204.ec2.internal" event="NodeHasSufficientPID" Jul 09 09:33:34.390062 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.390027 2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-204.ec2.internal" Jul 09 09:33:34.399091 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.399053 2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-204.ec2.internal" Jul 09 09:33:34.399091 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:34.399073 2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-204.ec2.internal\": node \"ip-10-0-135-204.ec2.internal\" not found" Jul 09 09:33:34.423909 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:34.423893 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-204.ec2.internal\" not found" Jul 09 09:33:34.478459 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.478421 2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-204.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-204.ec2.internal"] Jul 09 09:33:34.478515 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.478497 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 09:33:34.480138 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.480126 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-204.ec2.internal" event="NodeHasSufficientMemory" Jul 09 09:33:34.480197 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.480151 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-204.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 09:33:34.480197 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.480161 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-204.ec2.internal" event="NodeHasSufficientPID" Jul 09 09:33:34.481359 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.481348 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 09:33:34.481510 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.481497 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-204.ec2.internal" Jul 09 09:33:34.481548 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.481529 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 09:33:34.482718 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.482699 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-204.ec2.internal" event="NodeHasSufficientMemory" Jul 09 09:33:34.482788 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.482701 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-204.ec2.internal" event="NodeHasSufficientMemory" Jul 09 09:33:34.482788 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.482745 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-204.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 09:33:34.482788 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.482756 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-204.ec2.internal" event="NodeHasSufficientPID" Jul 09 09:33:34.482788 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.482728 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-204.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 09:33:34.482914 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.482800 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-204.ec2.internal" event="NodeHasSufficientPID" Jul 09 09:33:34.484107 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.484092 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-204.ec2.internal" Jul 09 09:33:34.484146 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.484120 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 09:33:34.485965 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.485950 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-204.ec2.internal" event="NodeHasSufficientMemory" Jul 09 09:33:34.486066 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.485975 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-204.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 09:33:34.486066 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.485986 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-204.ec2.internal" event="NodeHasSufficientPID" Jul 09 09:33:34.505975 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:34.505957 2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-204.ec2.internal\" not found" node="ip-10-0-135-204.ec2.internal" Jul 09 09:33:34.510119 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:34.510104 2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-204.ec2.internal\" not found" node="ip-10-0-135-204.ec2.internal" Jul 09 09:33:34.524974 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:34.524954 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-204.ec2.internal\" not found" Jul 09 09:33:34.535320 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.535303 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4386692e678e561186646b99422b104a-config\") pod \"kube-apiserver-proxy-ip-10-0-135-204.ec2.internal\" (UID: \"4386692e678e561186646b99422b104a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-204.ec2.internal" Jul 09 09:33:34.535384 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.535325 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f8f995f6fcf6abd8e78373eeaaaf3c86-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-204.ec2.internal\" (UID: \"f8f995f6fcf6abd8e78373eeaaaf3c86\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-204.ec2.internal" Jul 09 09:33:34.535384 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.535349 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f8f995f6fcf6abd8e78373eeaaaf3c86-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-204.ec2.internal\" (UID: \"f8f995f6fcf6abd8e78373eeaaaf3c86\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-204.ec2.internal" Jul 09 09:33:34.625707 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:34.625684 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-204.ec2.internal\" not found" Jul 09 09:33:34.636342 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.636322 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4386692e678e561186646b99422b104a-config\") pod \"kube-apiserver-proxy-ip-10-0-135-204.ec2.internal\" (UID: \"4386692e678e561186646b99422b104a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-204.ec2.internal" Jul 09 09:33:34.636403 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.636345 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f8f995f6fcf6abd8e78373eeaaaf3c86-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-204.ec2.internal\" (UID: \"f8f995f6fcf6abd8e78373eeaaaf3c86\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-204.ec2.internal" Jul 09 09:33:34.636403 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.636363 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f8f995f6fcf6abd8e78373eeaaaf3c86-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-204.ec2.internal\" (UID: \"f8f995f6fcf6abd8e78373eeaaaf3c86\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-204.ec2.internal" Jul 09 09:33:34.636403 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.636401 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f8f995f6fcf6abd8e78373eeaaaf3c86-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-204.ec2.internal\" (UID: \"f8f995f6fcf6abd8e78373eeaaaf3c86\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-204.ec2.internal" Jul 09 09:33:34.636556 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.636423 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4386692e678e561186646b99422b104a-config\") pod \"kube-apiserver-proxy-ip-10-0-135-204.ec2.internal\" (UID: \"4386692e678e561186646b99422b104a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-204.ec2.internal" Jul 09 09:33:34.636556 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.636477 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f8f995f6fcf6abd8e78373eeaaaf3c86-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-204.ec2.internal\" (UID: \"f8f995f6fcf6abd8e78373eeaaaf3c86\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-204.ec2.internal" Jul 09 09:33:34.726309 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:34.726259 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-204.ec2.internal\" not found" Jul 09 09:33:34.808759 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.808727 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-204.ec2.internal" Jul 09 09:33:34.813589 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:34.813575 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-204.ec2.internal" Jul 09 09:33:34.826894 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:34.826879 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-204.ec2.internal\" not found" Jul 09 09:33:34.927411 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:34.927388 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-204.ec2.internal\" not found" Jul 09 09:33:35.027910 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:35.027859 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-204.ec2.internal\" not found" Jul 09 09:33:35.037445 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.037427 2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 09:33:35.133556 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.133536 2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-204.ec2.internal" Jul 09 09:33:35.154010 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.153993 2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 09 09:33:35.155897 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.155884 2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-204.ec2.internal" Jul 09 09:33:35.164808 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.164792 2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 09 09:33:35.170011 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.169993 2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 09 09:33:35.170142 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.170119 2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 09 09:33:35.170244 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.170159 2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 09 09:33:35.213284 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.213260 2578 apiserver.go:52] "Watching apiserver" Jul 09 09:33:35.227508 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.227488 2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 09 09:33:35.227778 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.227760 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-nx4l4","openshift-multus/network-metrics-daemon-wsdxs","openshift-network-operator/iptables-alerter-r6npq","openshift-ovn-kubernetes/ovnkube-node-xf4sw","kube-system/global-pull-secret-syncer-sw9zq","kube-system/konnectivity-agent-gbmbr","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z76mc","openshift-image-registry/node-ca-qbtkh","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-204.ec2.internal","openshift-multus/multus-additional-cni-plugins-rspgp","openshift-multus/multus-clbg8","openshift-network-diagnostics/network-check-target-gqlhc","kube-system/kube-apiserver-proxy-ip-10-0-135-204.ec2.internal","openshift-cluster-node-tuning-operator/tuned-dnzdl"] Jul 09 09:33:35.229501 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.229488 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z76mc" Jul 09 09:33:35.230910 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.230888 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-clbg8" Jul 09 09:33:35.232104 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.232087 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-r6npq" Jul 09 09:33:35.232221 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.232201 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 09 09:33:35.232325 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.232210 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-w5tb6\"" Jul 09 09:33:35.232325 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.232319 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 09 09:33:35.232497 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.232482 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 09 09:33:35.232879 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.232861 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 09 09:33:35.233038 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.233018 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 09 09:33:35.233562 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.233540 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 09 09:33:35.233651 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.233639 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-4jjnp\"" Jul 09 09:33:35.233881 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.233863 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 09 09:33:35.234226 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.234205 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" Jul 09 09:33:35.234779 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.234756 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 09 09:33:35.234932 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.234910 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 09 09:33:35.235273 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.235252 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-w75b5\"" Jul 09 09:33:35.235932 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.235912 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 09 09:33:35.236274 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.236257 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 09 09:33:35.236519 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.236498 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sw9zq" Jul 09 09:33:35.236621 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:35.236599 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sw9zq" podUID="a0c24dc3-5e21-4537-966b-0c5f9e46f8d9" Jul 09 09:33:35.238039 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.238024 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gbmbr" Jul 09 09:33:35.238290 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.238267 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 09 09:33:35.238375 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.238356 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 09 09:33:35.238375 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.238370 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 09 09:33:35.238508 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.238392 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 09 09:33:35.238654 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.238640 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 09 09:33:35.238708 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.238669 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 09 09:33:35.239159 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.239143 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-dsj7c\"" Jul 09 09:33:35.239290 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.239273 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z56tr\" (UniqueName: \"kubernetes.io/projected/c7314861-eada-458d-bf61-21da88f8b1c8-kube-api-access-z56tr\") pod \"iptables-alerter-r6npq\" (UID: \"c7314861-eada-458d-bf61-21da88f8b1c8\") " pod="openshift-network-operator/iptables-alerter-r6npq" Jul 09 09:33:35.239374 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.239308 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1677420d-7376-45ad-ac67-20128aec5756-run-systemd\") pod \"ovnkube-node-xf4sw\" (UID: \"1677420d-7376-45ad-ac67-20128aec5756\") " pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" Jul 09 09:33:35.239374 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.239333 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1677420d-7376-45ad-ac67-20128aec5756-var-lib-openvswitch\") pod \"ovnkube-node-xf4sw\" (UID: \"1677420d-7376-45ad-ac67-20128aec5756\") " pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" Jul 09 09:33:35.239374 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.239368 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6e7da002-bfa6-4527-bfb9-ee3bc36677f1-etc-kubernetes\") pod \"multus-clbg8\" (UID: \"6e7da002-bfa6-4527-bfb9-ee3bc36677f1\") " pod="openshift-multus/multus-clbg8" Jul 09 09:33:35.239597 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.239392 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6e7da002-bfa6-4527-bfb9-ee3bc36677f1-hostroot\") pod \"multus-clbg8\" (UID: \"6e7da002-bfa6-4527-bfb9-ee3bc36677f1\") " pod="openshift-multus/multus-clbg8" Jul 09 09:33:35.239597 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.239416 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1677420d-7376-45ad-ac67-20128aec5756-host-slash\") pod \"ovnkube-node-xf4sw\" (UID: \"1677420d-7376-45ad-ac67-20128aec5756\") " pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" Jul 09 09:33:35.239597 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.239454 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1677420d-7376-45ad-ac67-20128aec5756-host-cni-netd\") pod \"ovnkube-node-xf4sw\" (UID: \"1677420d-7376-45ad-ac67-20128aec5756\") " pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" Jul 09 09:33:35.239597 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.239492 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f2d0ed74-687b-403d-9ad5-17ee9fc39d6a-registration-dir\") pod \"aws-ebs-csi-driver-node-z76mc\" (UID: \"f2d0ed74-687b-403d-9ad5-17ee9fc39d6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z76mc" Jul 09 09:33:35.239597 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.239505 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dnzdl" Jul 09 09:33:35.239597 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.239515 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1677420d-7376-45ad-ac67-20128aec5756-host-kubelet\") pod \"ovnkube-node-xf4sw\" (UID: \"1677420d-7376-45ad-ac67-20128aec5756\") " pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" Jul 09 09:33:35.239597 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.239556 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1677420d-7376-45ad-ac67-20128aec5756-node-log\") pod \"ovnkube-node-xf4sw\" (UID: \"1677420d-7376-45ad-ac67-20128aec5756\") " pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" Jul 09 09:33:35.239597 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.239581 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1677420d-7376-45ad-ac67-20128aec5756-host-cni-bin\") pod \"ovnkube-node-xf4sw\" (UID: \"1677420d-7376-45ad-ac67-20128aec5756\") " pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" Jul 09 09:33:35.239916 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.239604 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1677420d-7376-45ad-ac67-20128aec5756-ovn-node-metrics-cert\") pod \"ovnkube-node-xf4sw\" (UID: \"1677420d-7376-45ad-ac67-20128aec5756\") " pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" Jul 09 09:33:35.239916 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.239627 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5rnf\" (UniqueName: \"kubernetes.io/projected/1677420d-7376-45ad-ac67-20128aec5756-kube-api-access-c5rnf\") pod \"ovnkube-node-xf4sw\" (UID: \"1677420d-7376-45ad-ac67-20128aec5756\") " pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" Jul 09 09:33:35.239916 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.239649 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f2d0ed74-687b-403d-9ad5-17ee9fc39d6a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z76mc\" (UID: \"f2d0ed74-687b-403d-9ad5-17ee9fc39d6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z76mc" Jul 09 09:33:35.239916 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.239672 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1677420d-7376-45ad-ac67-20128aec5756-host-run-netns\") pod \"ovnkube-node-xf4sw\" (UID: \"1677420d-7376-45ad-ac67-20128aec5756\") " pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" Jul 09 09:33:35.239916 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.239697 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1677420d-7376-45ad-ac67-20128aec5756-run-ovn\") pod \"ovnkube-node-xf4sw\" (UID: \"1677420d-7376-45ad-ac67-20128aec5756\") " pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" Jul 09 09:33:35.239916 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.239718 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1677420d-7376-45ad-ac67-20128aec5756-ovnkube-script-lib\") pod \"ovnkube-node-xf4sw\" (UID: \"1677420d-7376-45ad-ac67-20128aec5756\") " pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" Jul 09 09:33:35.239916 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.239742 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c7314861-eada-458d-bf61-21da88f8b1c8-iptables-alerter-script\") pod \"iptables-alerter-r6npq\" (UID: \"c7314861-eada-458d-bf61-21da88f8b1c8\") " pod="openshift-network-operator/iptables-alerter-r6npq" Jul 09 09:33:35.239916 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.239764 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f2d0ed74-687b-403d-9ad5-17ee9fc39d6a-socket-dir\") pod \"aws-ebs-csi-driver-node-z76mc\" (UID: \"f2d0ed74-687b-403d-9ad5-17ee9fc39d6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z76mc" Jul 09 09:33:35.239916 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.239789 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f2d0ed74-687b-403d-9ad5-17ee9fc39d6a-device-dir\") pod \"aws-ebs-csi-driver-node-z76mc\" (UID: \"f2d0ed74-687b-403d-9ad5-17ee9fc39d6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z76mc" Jul 09 09:33:35.239916 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.239812 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f2d0ed74-687b-403d-9ad5-17ee9fc39d6a-sys-fs\") pod \"aws-ebs-csi-driver-node-z76mc\" (UID: \"f2d0ed74-687b-403d-9ad5-17ee9fc39d6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z76mc" Jul 09 09:33:35.239916 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.239867 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6e7da002-bfa6-4527-bfb9-ee3bc36677f1-multus-cni-dir\") pod \"multus-clbg8\" (UID: \"6e7da002-bfa6-4527-bfb9-ee3bc36677f1\") " pod="openshift-multus/multus-clbg8" Jul 09 09:33:35.239916 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.239904 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6e7da002-bfa6-4527-bfb9-ee3bc36677f1-multus-conf-dir\") pod \"multus-clbg8\" (UID: \"6e7da002-bfa6-4527-bfb9-ee3bc36677f1\") " pod="openshift-multus/multus-clbg8" Jul 09 09:33:35.240345 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.239919 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1677420d-7376-45ad-ac67-20128aec5756-systemd-units\") pod \"ovnkube-node-xf4sw\" (UID: \"1677420d-7376-45ad-ac67-20128aec5756\") " pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" Jul 09 09:33:35.240345 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.239941 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1677420d-7376-45ad-ac67-20128aec5756-host-run-ovn-kubernetes\") pod \"ovnkube-node-xf4sw\" (UID: \"1677420d-7376-45ad-ac67-20128aec5756\") " pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" Jul 09 09:33:35.240345 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.239982 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6e7da002-bfa6-4527-bfb9-ee3bc36677f1-cnibin\") pod \"multus-clbg8\" (UID: \"6e7da002-bfa6-4527-bfb9-ee3bc36677f1\") " pod="openshift-multus/multus-clbg8" Jul 09 09:33:35.240345 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.239997 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6e7da002-bfa6-4527-bfb9-ee3bc36677f1-multus-socket-dir-parent\") pod \"multus-clbg8\" (UID: \"6e7da002-bfa6-4527-bfb9-ee3bc36677f1\") " pod="openshift-multus/multus-clbg8" Jul 09 09:33:35.240345 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.240010 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6e7da002-bfa6-4527-bfb9-ee3bc36677f1-host-run-k8s-cni-cncf-io\") pod \"multus-clbg8\" (UID: \"6e7da002-bfa6-4527-bfb9-ee3bc36677f1\") " pod="openshift-multus/multus-clbg8" Jul 09 09:33:35.240345 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.240036 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6e7da002-bfa6-4527-bfb9-ee3bc36677f1-host-var-lib-cni-multus\") pod \"multus-clbg8\" (UID: \"6e7da002-bfa6-4527-bfb9-ee3bc36677f1\") " pod="openshift-multus/multus-clbg8" Jul 09 09:33:35.240345 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.240049 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c7314861-eada-458d-bf61-21da88f8b1c8-host-slash\") pod \"iptables-alerter-r6npq\" (UID: \"c7314861-eada-458d-bf61-21da88f8b1c8\") " pod="openshift-network-operator/iptables-alerter-r6npq" Jul 09 09:33:35.240345 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.240072 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1677420d-7376-45ad-ac67-20128aec5756-run-openvswitch\") pod \"ovnkube-node-xf4sw\" (UID: \"1677420d-7376-45ad-ac67-20128aec5756\") " pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" Jul 09 09:33:35.240345 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.240115 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1677420d-7376-45ad-ac67-20128aec5756-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xf4sw\" (UID: \"1677420d-7376-45ad-ac67-20128aec5756\") " pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" Jul 09 09:33:35.240345 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.240131 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f2d0ed74-687b-403d-9ad5-17ee9fc39d6a-etc-selinux\") pod \"aws-ebs-csi-driver-node-z76mc\" (UID: \"f2d0ed74-687b-403d-9ad5-17ee9fc39d6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z76mc" Jul 09 09:33:35.240345 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.240144 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6e7da002-bfa6-4527-bfb9-ee3bc36677f1-system-cni-dir\") pod \"multus-clbg8\" (UID: \"6e7da002-bfa6-4527-bfb9-ee3bc36677f1\") " pod="openshift-multus/multus-clbg8" Jul 09 09:33:35.240345 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.240178 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6e7da002-bfa6-4527-bfb9-ee3bc36677f1-host-run-netns\") pod \"multus-clbg8\" (UID: \"6e7da002-bfa6-4527-bfb9-ee3bc36677f1\") " pod="openshift-multus/multus-clbg8" Jul 09 09:33:35.240345 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.240200 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6e7da002-bfa6-4527-bfb9-ee3bc36677f1-host-var-lib-kubelet\") pod \"multus-clbg8\" (UID: \"6e7da002-bfa6-4527-bfb9-ee3bc36677f1\") " pod="openshift-multus/multus-clbg8" Jul 09 09:33:35.240345 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.240204 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 09 09:33:35.240345 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.240234 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvbxg\" (UniqueName: \"kubernetes.io/projected/6e7da002-bfa6-4527-bfb9-ee3bc36677f1-kube-api-access-vvbxg\") pod \"multus-clbg8\" (UID: \"6e7da002-bfa6-4527-bfb9-ee3bc36677f1\") " pod="openshift-multus/multus-clbg8" Jul 09 09:33:35.240345 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.240271 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1677420d-7376-45ad-ac67-20128aec5756-env-overrides\") pod \"ovnkube-node-xf4sw\" (UID: \"1677420d-7376-45ad-ac67-20128aec5756\") " pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" Jul 09 09:33:35.240345 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.240296 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6e7da002-bfa6-4527-bfb9-ee3bc36677f1-host-var-lib-cni-bin\") pod \"multus-clbg8\" (UID: \"6e7da002-bfa6-4527-bfb9-ee3bc36677f1\") " pod="openshift-multus/multus-clbg8" Jul 09 09:33:35.240879 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.240331 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1677420d-7376-45ad-ac67-20128aec5756-etc-openvswitch\") pod \"ovnkube-node-xf4sw\" (UID: \"1677420d-7376-45ad-ac67-20128aec5756\") " pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" Jul 09 09:33:35.240879 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.240361 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1677420d-7376-45ad-ac67-20128aec5756-log-socket\") pod \"ovnkube-node-xf4sw\" (UID: \"1677420d-7376-45ad-ac67-20128aec5756\") " pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" Jul 09 09:33:35.240879 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.240380 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 09 09:33:35.240879 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.240388 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1677420d-7376-45ad-ac67-20128aec5756-ovnkube-config\") pod \"ovnkube-node-xf4sw\" (UID: \"1677420d-7376-45ad-ac67-20128aec5756\") " pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" Jul 09 09:33:35.240879 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.240412 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6e7da002-bfa6-4527-bfb9-ee3bc36677f1-host-run-multus-certs\") pod \"multus-clbg8\" (UID: \"6e7da002-bfa6-4527-bfb9-ee3bc36677f1\") " pod="openshift-multus/multus-clbg8" Jul 09 09:33:35.240879 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.240460 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n8vt\" (UniqueName: \"kubernetes.io/projected/f2d0ed74-687b-403d-9ad5-17ee9fc39d6a-kube-api-access-5n8vt\") pod \"aws-ebs-csi-driver-node-z76mc\" (UID: \"f2d0ed74-687b-403d-9ad5-17ee9fc39d6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z76mc" Jul 09 09:33:35.240879 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.240487 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6e7da002-bfa6-4527-bfb9-ee3bc36677f1-os-release\") pod \"multus-clbg8\" (UID: \"6e7da002-bfa6-4527-bfb9-ee3bc36677f1\") " pod="openshift-multus/multus-clbg8" Jul 09 09:33:35.240879 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.240507 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-sqpp8\"" Jul 09 09:33:35.240879 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.240511 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6e7da002-bfa6-4527-bfb9-ee3bc36677f1-cni-binary-copy\") pod \"multus-clbg8\" (UID: \"6e7da002-bfa6-4527-bfb9-ee3bc36677f1\") " pod="openshift-multus/multus-clbg8" Jul 09 09:33:35.240879 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.240533 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6e7da002-bfa6-4527-bfb9-ee3bc36677f1-multus-daemon-config\") pod \"multus-clbg8\" (UID: \"6e7da002-bfa6-4527-bfb9-ee3bc36677f1\") " pod="openshift-multus/multus-clbg8" Jul 09 09:33:35.241226 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.241211 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qbtkh" Jul 09 09:33:35.241383 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.241318 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-08 09:28:35 +0000 UTC" deadline="2028-02-21 10:20:25.928906215 +0000 UTC" Jul 09 09:33:35.241462 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.241384 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14208h46m50.687525725s" Jul 09 09:33:35.241582 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.241565 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zb4dg\"" Jul 09 09:33:35.242106 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.242094 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 09 09:33:35.242250 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.242240 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 09 09:33:35.242613 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.242601 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wsdxs" Jul 09 09:33:35.242663 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:35.242653 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wsdxs" podUID="c0652b01-6ddc-4820-8fbe-c2e5b643f663" Jul 09 09:33:35.243285 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.243268 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 09 09:33:35.243547 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.243531 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 09 09:33:35.243623 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.243605 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 09 09:33:35.243675 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.243657 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-c6ptw\"" Jul 09 09:33:35.244647 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.244631 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rspgp" Jul 09 09:33:35.245870 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.245857 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nx4l4" Jul 09 09:33:35.247090 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.247075 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gqlhc" Jul 09 09:33:35.247155 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:35.247126 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gqlhc" podUID="53e2e8a5-34d5-4317-9733-143a53880b9b" Jul 09 09:33:35.248937 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.248922 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 09 09:33:35.249007 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.248972 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 09 09:33:35.249094 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.249081 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 09 09:33:35.249176 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.249162 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 09 09:33:35.249232 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.249169 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2vvqv\"" Jul 09 09:33:35.249322 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.249305 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xcrp4\"" Jul 09 09:33:35.249359 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.249307 2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 09 09:33:35.279028 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.278986 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-z96w2" Jul 09 09:33:35.283797 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.283783 2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 09:33:35.287901 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.287886 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-z96w2" Jul 09 09:33:35.334344 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.334320 2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 09 09:33:35.341543 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.341526 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqtj4\" (UniqueName: \"kubernetes.io/projected/32307b23-721f-4ed0-add4-542ed80a29d8-kube-api-access-bqtj4\") pod \"node-resolver-nx4l4\" (UID: \"32307b23-721f-4ed0-add4-542ed80a29d8\") " pod="openshift-dns/node-resolver-nx4l4" Jul 09 09:33:35.341611 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.341552 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46a45a21-2ddc-4b0b-9988-911cde1598f7-etc-kubernetes\") pod \"tuned-dnzdl\" (UID: \"46a45a21-2ddc-4b0b-9988-911cde1598f7\") " pod="openshift-cluster-node-tuning-operator/tuned-dnzdl" Jul 09 09:33:35.341611 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.341572 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f2d0ed74-687b-403d-9ad5-17ee9fc39d6a-socket-dir\") pod \"aws-ebs-csi-driver-node-z76mc\" (UID: \"f2d0ed74-687b-403d-9ad5-17ee9fc39d6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z76mc" Jul 09 09:33:35.341611 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.341588 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6e7da002-bfa6-4527-bfb9-ee3bc36677f1-multus-cni-dir\") pod \"multus-clbg8\" (UID: \"6e7da002-bfa6-4527-bfb9-ee3bc36677f1\") " pod="openshift-multus/multus-clbg8" Jul 09 09:33:35.341717 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.341631 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6e7da002-bfa6-4527-bfb9-ee3bc36677f1-multus-conf-dir\") pod \"multus-clbg8\" (UID: \"6e7da002-bfa6-4527-bfb9-ee3bc36677f1\") " pod="openshift-multus/multus-clbg8" Jul 09 09:33:35.341760 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.341710 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6e7da002-bfa6-4527-bfb9-ee3bc36677f1-multus-cni-dir\") pod \"multus-clbg8\" (UID: \"6e7da002-bfa6-4527-bfb9-ee3bc36677f1\") " pod="openshift-multus/multus-clbg8" Jul 09 09:33:35.341760 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.341725 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1677420d-7376-45ad-ac67-20128aec5756-systemd-units\") pod \"ovnkube-node-xf4sw\" (UID: \"1677420d-7376-45ad-ac67-20128aec5756\") " pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" Jul 09 09:33:35.341760 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.341749 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1677420d-7376-45ad-ac67-20128aec5756-host-run-ovn-kubernetes\") pod \"ovnkube-node-xf4sw\" (UID: \"1677420d-7376-45ad-ac67-20128aec5756\") " pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" Jul 09 09:33:35.341760 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.341753 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1677420d-7376-45ad-ac67-20128aec5756-systemd-units\") pod \"ovnkube-node-xf4sw\" (UID: \"1677420d-7376-45ad-ac67-20128aec5756\") " pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" Jul 09 09:33:35.341909 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.341711 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6e7da002-bfa6-4527-bfb9-ee3bc36677f1-multus-conf-dir\") pod \"multus-clbg8\" (UID: \"6e7da002-bfa6-4527-bfb9-ee3bc36677f1\") " pod="openshift-multus/multus-clbg8" Jul 09 09:33:35.341909 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.341766 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlvnh\" (UniqueName: \"kubernetes.io/projected/f8587efa-7f0b-4a11-b03f-e08b942aa1f1-kube-api-access-dlvnh\") pod \"node-ca-qbtkh\" (UID: \"f8587efa-7f0b-4a11-b03f-e08b942aa1f1\") " pod="openshift-image-registry/node-ca-qbtkh" Jul 09 09:33:35.341909 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.341783 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/32307b23-721f-4ed0-add4-542ed80a29d8-hosts-file\") pod \"node-resolver-nx4l4\" (UID: \"32307b23-721f-4ed0-add4-542ed80a29d8\") " pod="openshift-dns/node-resolver-nx4l4" Jul 09 09:33:35.341909 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.341789 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1677420d-7376-45ad-ac67-20128aec5756-host-run-ovn-kubernetes\") pod \"ovnkube-node-xf4sw\" (UID: \"1677420d-7376-45ad-ac67-20128aec5756\") " pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" Jul 09 09:33:35.341909 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.341810 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6e7da002-bfa6-4527-bfb9-ee3bc36677f1-multus-socket-dir-parent\") pod \"multus-clbg8\" (UID: \"6e7da002-bfa6-4527-bfb9-ee3bc36677f1\") " pod="openshift-multus/multus-clbg8" Jul 09 09:33:35.341909 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.341842 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6e7da002-bfa6-4527-bfb9-ee3bc36677f1-host-run-k8s-cni-cncf-io\") pod \"multus-clbg8\" (UID: \"6e7da002-bfa6-4527-bfb9-ee3bc36677f1\") " pod="openshift-multus/multus-clbg8" Jul 09 09:33:35.341909 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.341865 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6e7da002-bfa6-4527-bfb9-ee3bc36677f1-host-var-lib-cni-multus\") pod \"multus-clbg8\" (UID: \"6e7da002-bfa6-4527-bfb9-ee3bc36677f1\") " pod="openshift-multus/multus-clbg8" Jul 09 09:33:35.341909 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.341883 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6e7da002-bfa6-4527-bfb9-ee3bc36677f1-host-run-k8s-cni-cncf-io\") pod \"multus-clbg8\" (UID: \"6e7da002-bfa6-4527-bfb9-ee3bc36677f1\") " pod="openshift-multus/multus-clbg8" Jul 09 09:33:35.341909 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.341893 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a0c24dc3-5e21-4537-966b-0c5f9e46f8d9-kubelet-config\") pod \"global-pull-secret-syncer-sw9zq\" (UID: \"a0c24dc3-5e21-4537-966b-0c5f9e46f8d9\") " pod="kube-system/global-pull-secret-syncer-sw9zq" Jul 09 09:33:35.341909 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.341915 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6e7da002-bfa6-4527-bfb9-ee3bc36677f1-host-var-lib-cni-multus\") pod \"multus-clbg8\" (UID: \"6e7da002-bfa6-4527-bfb9-ee3bc36677f1\") " pod="openshift-multus/multus-clbg8" Jul 09 09:33:35.342221 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.341918 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dc248a4e-4aa6-4ea4-81b7-439e669b56c1-konnectivity-ca\") pod \"konnectivity-agent-gbmbr\" (UID: \"dc248a4e-4aa6-4ea4-81b7-439e669b56c1\") " pod="kube-system/konnectivity-agent-gbmbr" Jul 09 09:33:35.342221 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.341921 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6e7da002-bfa6-4527-bfb9-ee3bc36677f1-multus-socket-dir-parent\") pod \"multus-clbg8\" (UID: \"6e7da002-bfa6-4527-bfb9-ee3bc36677f1\") " pod="openshift-multus/multus-clbg8" Jul 09 09:33:35.342221 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.341940 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/46a45a21-2ddc-4b0b-9988-911cde1598f7-etc-sysconfig\") pod \"tuned-dnzdl\" (UID: \"46a45a21-2ddc-4b0b-9988-911cde1598f7\") " pod="openshift-cluster-node-tuning-operator/tuned-dnzdl" Jul 09 09:33:35.342221 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.341961 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/65d9652f-67de-405c-8e07-aa737e9205d8-cni-binary-copy\") pod \"multus-additional-cni-plugins-rspgp\" (UID: \"65d9652f-67de-405c-8e07-aa737e9205d8\") " pod="openshift-multus/multus-additional-cni-plugins-rspgp" Jul 09 09:33:35.342221 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.341981 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6e7da002-bfa6-4527-bfb9-ee3bc36677f1-system-cni-dir\") pod \"multus-clbg8\" (UID: \"6e7da002-bfa6-4527-bfb9-ee3bc36677f1\") " pod="openshift-multus/multus-clbg8" Jul 09 09:33:35.342221 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.341995 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6e7da002-bfa6-4527-bfb9-ee3bc36677f1-host-run-netns\") pod \"multus-clbg8\" (UID: \"6e7da002-bfa6-4527-bfb9-ee3bc36677f1\") " pod="openshift-multus/multus-clbg8" Jul 09 09:33:35.342221 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.342016 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvbxg\" (UniqueName: \"kubernetes.io/projected/6e7da002-bfa6-4527-bfb9-ee3bc36677f1-kube-api-access-vvbxg\") pod \"multus-clbg8\" (UID: \"6e7da002-bfa6-4527-bfb9-ee3bc36677f1\") " pod="openshift-multus/multus-clbg8" Jul 09 09:33:35.342221 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.342028 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6e7da002-bfa6-4527-bfb9-ee3bc36677f1-system-cni-dir\") pod \"multus-clbg8\" (UID: \"6e7da002-bfa6-4527-bfb9-ee3bc36677f1\") " pod="openshift-multus/multus-clbg8" Jul 09 09:33:35.342221 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.342033 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1677420d-7376-45ad-ac67-20128aec5756-env-overrides\") pod \"ovnkube-node-xf4sw\" (UID: \"1677420d-7376-45ad-ac67-20128aec5756\") " pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" Jul 09 09:33:35.342221 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.342054 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6e7da002-bfa6-4527-bfb9-ee3bc36677f1-host-run-netns\") pod \"multus-clbg8\" (UID: \"6e7da002-bfa6-4527-bfb9-ee3bc36677f1\") " pod="openshift-multus/multus-clbg8" Jul 09 09:33:35.342221 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.342072 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/46a45a21-2ddc-4b0b-9988-911cde1598f7-etc-sysctl-conf\") pod \"tuned-dnzdl\" (UID: \"46a45a21-2ddc-4b0b-9988-911cde1598f7\") " pod="openshift-cluster-node-tuning-operator/tuned-dnzdl" Jul 09 09:33:35.342221 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.342090 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6e7da002-bfa6-4527-bfb9-ee3bc36677f1-host-var-lib-cni-bin\") pod \"multus-clbg8\" (UID: \"6e7da002-bfa6-4527-bfb9-ee3bc36677f1\") " pod="openshift-multus/multus-clbg8" Jul 09 09:33:35.342221 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.342107 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1677420d-7376-45ad-ac67-20128aec5756-etc-openvswitch\") pod \"ovnkube-node-xf4sw\" (UID: \"1677420d-7376-45ad-ac67-20128aec5756\") " pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" Jul 09 09:33:35.342221 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.342121 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n8vt\" (UniqueName: \"kubernetes.io/projected/f2d0ed74-687b-403d-9ad5-17ee9fc39d6a-kube-api-access-5n8vt\") pod \"aws-ebs-csi-driver-node-z76mc\" (UID: \"f2d0ed74-687b-403d-9ad5-17ee9fc39d6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z76mc" Jul 09 09:33:35.342221 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.342143 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6e7da002-bfa6-4527-bfb9-ee3bc36677f1-multus-daemon-config\") pod \"multus-clbg8\" (UID: \"6e7da002-bfa6-4527-bfb9-ee3bc36677f1\") " pod="openshift-multus/multus-clbg8" Jul 09 09:33:35.342221 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.342169 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z56tr\" (UniqueName: \"kubernetes.io/projected/c7314861-eada-458d-bf61-21da88f8b1c8-kube-api-access-z56tr\") pod \"iptables-alerter-r6npq\" (UID: \"c7314861-eada-458d-bf61-21da88f8b1c8\") " pod="openshift-network-operator/iptables-alerter-r6npq" Jul 09 09:33:35.342221 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.342170 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6e7da002-bfa6-4527-bfb9-ee3bc36677f1-host-var-lib-cni-bin\") pod \"multus-clbg8\" (UID: \"6e7da002-bfa6-4527-bfb9-ee3bc36677f1\") " pod="openshift-multus/multus-clbg8" Jul 09 09:33:35.342999 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.342185 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1677420d-7376-45ad-ac67-20128aec5756-etc-openvswitch\") pod \"ovnkube-node-xf4sw\" (UID: \"1677420d-7376-45ad-ac67-20128aec5756\") " pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" Jul 09 09:33:35.342999 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.342204 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1677420d-7376-45ad-ac67-20128aec5756-run-systemd\") pod \"ovnkube-node-xf4sw\" (UID: \"1677420d-7376-45ad-ac67-20128aec5756\") " pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" Jul 09 09:33:35.342999 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.342241 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1677420d-7376-45ad-ac67-20128aec5756-run-systemd\") pod \"ovnkube-node-xf4sw\" (UID: \"1677420d-7376-45ad-ac67-20128aec5756\") " pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" Jul 09 09:33:35.342999 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.342269 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/46a45a21-2ddc-4b0b-9988-911cde1598f7-etc-tuned\") pod \"tuned-dnzdl\" (UID: \"46a45a21-2ddc-4b0b-9988-911cde1598f7\") " pod="openshift-cluster-node-tuning-operator/tuned-dnzdl" Jul 09 09:33:35.342999 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.342292 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6e7da002-bfa6-4527-bfb9-ee3bc36677f1-hostroot\") pod \"multus-clbg8\" (UID: \"6e7da002-bfa6-4527-bfb9-ee3bc36677f1\") " pod="openshift-multus/multus-clbg8" Jul 09 09:33:35.342999 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.342314 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0652b01-6ddc-4820-8fbe-c2e5b643f663-metrics-certs\") pod \"network-metrics-daemon-wsdxs\" (UID: \"c0652b01-6ddc-4820-8fbe-c2e5b643f663\") " pod="openshift-multus/network-metrics-daemon-wsdxs" Jul 09 09:33:35.342999 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.342339 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/65d9652f-67de-405c-8e07-aa737e9205d8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rspgp\" (UID: \"65d9652f-67de-405c-8e07-aa737e9205d8\") " pod="openshift-multus/multus-additional-cni-plugins-rspgp" Jul 09 09:33:35.342999 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.342340 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6e7da002-bfa6-4527-bfb9-ee3bc36677f1-hostroot\") pod \"multus-clbg8\" (UID: \"6e7da002-bfa6-4527-bfb9-ee3bc36677f1\") " pod="openshift-multus/multus-clbg8" Jul 09 09:33:35.342999 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.342367 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1677420d-7376-45ad-ac67-20128aec5756-host-kubelet\") pod \"ovnkube-node-xf4sw\" (UID: \"1677420d-7376-45ad-ac67-20128aec5756\") " pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" Jul 09 09:33:35.342999 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.342393 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1677420d-7376-45ad-ac67-20128aec5756-ovn-node-metrics-cert\") pod \"ovnkube-node-xf4sw\" (UID: \"1677420d-7376-45ad-ac67-20128aec5756\") " pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" Jul 09 09:33:35.342999 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.342414 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dc248a4e-4aa6-4ea4-81b7-439e669b56c1-agent-certs\") pod \"konnectivity-agent-gbmbr\" (UID: \"dc248a4e-4aa6-4ea4-81b7-439e669b56c1\") " pod="kube-system/konnectivity-agent-gbmbr" Jul 09 09:33:35.342999 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.342466 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f2d0ed74-687b-403d-9ad5-17ee9fc39d6a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z76mc\" (UID: \"f2d0ed74-687b-403d-9ad5-17ee9fc39d6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z76mc" Jul 09 09:33:35.342999 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.342491 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f8587efa-7f0b-4a11-b03f-e08b942aa1f1-serviceca\") pod \"node-ca-qbtkh\" (UID: \"f8587efa-7f0b-4a11-b03f-e08b942aa1f1\") " pod="openshift-image-registry/node-ca-qbtkh" Jul 09 09:33:35.342999 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.342500 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f2d0ed74-687b-403d-9ad5-17ee9fc39d6a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z76mc\" (UID: \"f2d0ed74-687b-403d-9ad5-17ee9fc39d6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z76mc" Jul 09 09:33:35.342999 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.342468 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1677420d-7376-45ad-ac67-20128aec5756-host-kubelet\") pod \"ovnkube-node-xf4sw\" (UID: \"1677420d-7376-45ad-ac67-20128aec5756\") " pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" Jul 09 09:33:35.342999 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.342515 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c7314861-eada-458d-bf61-21da88f8b1c8-iptables-alerter-script\") pod \"iptables-alerter-r6npq\" (UID: \"c7314861-eada-458d-bf61-21da88f8b1c8\") " pod="openshift-network-operator/iptables-alerter-r6npq" Jul 09 09:33:35.342999 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.342538 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f2d0ed74-687b-403d-9ad5-17ee9fc39d6a-device-dir\") pod \"aws-ebs-csi-driver-node-z76mc\" (UID: \"f2d0ed74-687b-403d-9ad5-17ee9fc39d6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z76mc" Jul 09 09:33:35.343567 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.342554 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f2d0ed74-687b-403d-9ad5-17ee9fc39d6a-sys-fs\") pod \"aws-ebs-csi-driver-node-z76mc\" (UID: \"f2d0ed74-687b-403d-9ad5-17ee9fc39d6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z76mc" Jul 09 09:33:35.343567 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.342569 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1677420d-7376-45ad-ac67-20128aec5756-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xf4sw\" (UID: \"1677420d-7376-45ad-ac67-20128aec5756\") " pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" Jul 09 09:33:35.343567 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.342586 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/46a45a21-2ddc-4b0b-9988-911cde1598f7-etc-sysctl-d\") pod \"tuned-dnzdl\" (UID: \"46a45a21-2ddc-4b0b-9988-911cde1598f7\") " pod="openshift-cluster-node-tuning-operator/tuned-dnzdl" Jul 09 09:33:35.343567 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.342602 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/46a45a21-2ddc-4b0b-9988-911cde1598f7-lib-modules\") pod \"tuned-dnzdl\" (UID: \"46a45a21-2ddc-4b0b-9988-911cde1598f7\") " pod="openshift-cluster-node-tuning-operator/tuned-dnzdl" Jul 09 09:33:35.343567 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.342622 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/46a45a21-2ddc-4b0b-9988-911cde1598f7-host\") pod \"tuned-dnzdl\" (UID: \"46a45a21-2ddc-4b0b-9988-911cde1598f7\") " pod="openshift-cluster-node-tuning-operator/tuned-dnzdl" Jul 09 09:33:35.343567 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.342640 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f2d0ed74-687b-403d-9ad5-17ee9fc39d6a-sys-fs\") pod \"aws-ebs-csi-driver-node-z76mc\" (UID: \"f2d0ed74-687b-403d-9ad5-17ee9fc39d6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z76mc" Jul 09 09:33:35.343567 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.342659 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/46a45a21-2ddc-4b0b-9988-911cde1598f7-tmp\") pod \"tuned-dnzdl\" (UID: \"46a45a21-2ddc-4b0b-9988-911cde1598f7\") " pod="openshift-cluster-node-tuning-operator/tuned-dnzdl" Jul 09 09:33:35.343567 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.342667 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f2d0ed74-687b-403d-9ad5-17ee9fc39d6a-socket-dir\") pod \"aws-ebs-csi-driver-node-z76mc\" (UID: \"f2d0ed74-687b-403d-9ad5-17ee9fc39d6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z76mc" Jul 09 09:33:35.343567 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.342685 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6e7da002-bfa6-4527-bfb9-ee3bc36677f1-cnibin\") pod \"multus-clbg8\" (UID: \"6e7da002-bfa6-4527-bfb9-ee3bc36677f1\") " pod="openshift-multus/multus-clbg8" Jul 09 09:33:35.343567 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.342691 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f2d0ed74-687b-403d-9ad5-17ee9fc39d6a-device-dir\") pod \"aws-ebs-csi-driver-node-z76mc\" (UID: \"f2d0ed74-687b-403d-9ad5-17ee9fc39d6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z76mc" Jul 09 09:33:35.343567 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.342710 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c7314861-eada-458d-bf61-21da88f8b1c8-host-slash\") pod \"iptables-alerter-r6npq\" (UID: \"c7314861-eada-458d-bf61-21da88f8b1c8\") " pod="openshift-network-operator/iptables-alerter-r6npq" Jul 09 09:33:35.343567 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.342715 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1677420d-7376-45ad-ac67-20128aec5756-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xf4sw\" (UID: \"1677420d-7376-45ad-ac67-20128aec5756\") " pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" Jul 09 09:33:35.343567 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.342742 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c7314861-eada-458d-bf61-21da88f8b1c8-host-slash\") pod \"iptables-alerter-r6npq\" (UID: \"c7314861-eada-458d-bf61-21da88f8b1c8\") " pod="openshift-network-operator/iptables-alerter-r6npq" Jul 09 09:33:35.343567 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.342746 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6e7da002-bfa6-4527-bfb9-ee3bc36677f1-multus-daemon-config\") pod \"multus-clbg8\" (UID: \"6e7da002-bfa6-4527-bfb9-ee3bc36677f1\") " pod="openshift-multus/multus-clbg8" Jul 09 09:33:35.343567 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.342742 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1677420d-7376-45ad-ac67-20128aec5756-run-openvswitch\") pod \"ovnkube-node-xf4sw\" (UID: \"1677420d-7376-45ad-ac67-20128aec5756\") " pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" Jul 09 09:33:35.343567 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.342765 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6e7da002-bfa6-4527-bfb9-ee3bc36677f1-cnibin\") pod \"multus-clbg8\" (UID: \"6e7da002-bfa6-4527-bfb9-ee3bc36677f1\") " pod="openshift-multus/multus-clbg8" Jul 09 09:33:35.343567 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.342775 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1677420d-7376-45ad-ac67-20128aec5756-run-openvswitch\") pod \"ovnkube-node-xf4sw\" (UID: \"1677420d-7376-45ad-ac67-20128aec5756\") " pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" Jul 09 09:33:35.344279 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.342785 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32307b23-721f-4ed0-add4-542ed80a29d8-tmp-dir\") pod \"node-resolver-nx4l4\" (UID: \"32307b23-721f-4ed0-add4-542ed80a29d8\") " pod="openshift-dns/node-resolver-nx4l4" Jul 09 09:33:35.344279 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.342809 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87rbm\" (UniqueName: \"kubernetes.io/projected/46a45a21-2ddc-4b0b-9988-911cde1598f7-kube-api-access-87rbm\") pod \"tuned-dnzdl\" (UID: \"46a45a21-2ddc-4b0b-9988-911cde1598f7\") " pod="openshift-cluster-node-tuning-operator/tuned-dnzdl" Jul 09 09:33:35.344279 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.342834 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/65d9652f-67de-405c-8e07-aa737e9205d8-cnibin\") pod \"multus-additional-cni-plugins-rspgp\" (UID: \"65d9652f-67de-405c-8e07-aa737e9205d8\") " pod="openshift-multus/multus-additional-cni-plugins-rspgp" Jul 09 09:33:35.344279 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.342873 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f2d0ed74-687b-403d-9ad5-17ee9fc39d6a-etc-selinux\") pod \"aws-ebs-csi-driver-node-z76mc\" (UID: \"f2d0ed74-687b-403d-9ad5-17ee9fc39d6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z76mc" Jul 09 09:33:35.344279 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.342905 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6e7da002-bfa6-4527-bfb9-ee3bc36677f1-host-var-lib-kubelet\") pod \"multus-clbg8\" (UID: \"6e7da002-bfa6-4527-bfb9-ee3bc36677f1\") " pod="openshift-multus/multus-clbg8" Jul 09 09:33:35.344279 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.342933 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a0c24dc3-5e21-4537-966b-0c5f9e46f8d9-original-pull-secret\") pod \"global-pull-secret-syncer-sw9zq\" (UID: \"a0c24dc3-5e21-4537-966b-0c5f9e46f8d9\") " pod="kube-system/global-pull-secret-syncer-sw9zq" Jul 09 09:33:35.344279 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.342964 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1677420d-7376-45ad-ac67-20128aec5756-log-socket\") pod \"ovnkube-node-xf4sw\" (UID: \"1677420d-7376-45ad-ac67-20128aec5756\") " pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" Jul 09 09:33:35.344279 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.342967 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6e7da002-bfa6-4527-bfb9-ee3bc36677f1-host-var-lib-kubelet\") pod \"multus-clbg8\" (UID: \"6e7da002-bfa6-4527-bfb9-ee3bc36677f1\") " pod="openshift-multus/multus-clbg8" Jul 09 09:33:35.344279 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.342974 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f2d0ed74-687b-403d-9ad5-17ee9fc39d6a-etc-selinux\") pod \"aws-ebs-csi-driver-node-z76mc\" (UID: \"f2d0ed74-687b-403d-9ad5-17ee9fc39d6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z76mc" Jul 09 09:33:35.344279 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.342988 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1677420d-7376-45ad-ac67-20128aec5756-ovnkube-config\") pod \"ovnkube-node-xf4sw\" (UID: \"1677420d-7376-45ad-ac67-20128aec5756\") " pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" Jul 09 09:33:35.344279 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.343000 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c7314861-eada-458d-bf61-21da88f8b1c8-iptables-alerter-script\") pod \"iptables-alerter-r6npq\" (UID: \"c7314861-eada-458d-bf61-21da88f8b1c8\") " pod="openshift-network-operator/iptables-alerter-r6npq" Jul 09 09:33:35.344279 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.343015 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1677420d-7376-45ad-ac67-20128aec5756-log-socket\") pod \"ovnkube-node-xf4sw\" (UID: \"1677420d-7376-45ad-ac67-20128aec5756\") " pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" Jul 09 09:33:35.344279 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.343046 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a0c24dc3-5e21-4537-966b-0c5f9e46f8d9-dbus\") pod \"global-pull-secret-syncer-sw9zq\" (UID: \"a0c24dc3-5e21-4537-966b-0c5f9e46f8d9\") " pod="kube-system/global-pull-secret-syncer-sw9zq" Jul 09 09:33:35.344279 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.343091 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/46a45a21-2ddc-4b0b-9988-911cde1598f7-etc-systemd\") pod \"tuned-dnzdl\" (UID: \"46a45a21-2ddc-4b0b-9988-911cde1598f7\") " pod="openshift-cluster-node-tuning-operator/tuned-dnzdl" Jul 09 09:33:35.344279 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.343107 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/46a45a21-2ddc-4b0b-9988-911cde1598f7-sys\") pod \"tuned-dnzdl\" (UID: \"46a45a21-2ddc-4b0b-9988-911cde1598f7\") " pod="openshift-cluster-node-tuning-operator/tuned-dnzdl" Jul 09 09:33:35.344279 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.343140 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/65d9652f-67de-405c-8e07-aa737e9205d8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rspgp\" (UID: \"65d9652f-67de-405c-8e07-aa737e9205d8\") " pod="openshift-multus/multus-additional-cni-plugins-rspgp" Jul 09 09:33:35.344279 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.343178 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6e7da002-bfa6-4527-bfb9-ee3bc36677f1-host-run-multus-certs\") pod \"multus-clbg8\" (UID: \"6e7da002-bfa6-4527-bfb9-ee3bc36677f1\") " pod="openshift-multus/multus-clbg8" Jul 09 09:33:35.344757 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.343194 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6e7da002-bfa6-4527-bfb9-ee3bc36677f1-os-release\") pod \"multus-clbg8\" (UID: \"6e7da002-bfa6-4527-bfb9-ee3bc36677f1\") " pod="openshift-multus/multus-clbg8" Jul 09 09:33:35.344757 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.343219 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6e7da002-bfa6-4527-bfb9-ee3bc36677f1-cni-binary-copy\") pod \"multus-clbg8\" (UID: \"6e7da002-bfa6-4527-bfb9-ee3bc36677f1\") " pod="openshift-multus/multus-clbg8" Jul 09 09:33:35.344757 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.343291 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6e7da002-bfa6-4527-bfb9-ee3bc36677f1-host-run-multus-certs\") pod \"multus-clbg8\" (UID: \"6e7da002-bfa6-4527-bfb9-ee3bc36677f1\") " pod="openshift-multus/multus-clbg8" Jul 09 09:33:35.344757 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.343291 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1677420d-7376-45ad-ac67-20128aec5756-var-lib-openvswitch\") pod \"ovnkube-node-xf4sw\" (UID: \"1677420d-7376-45ad-ac67-20128aec5756\") " pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" Jul 09 09:33:35.344757 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.343320 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6e7da002-bfa6-4527-bfb9-ee3bc36677f1-os-release\") pod \"multus-clbg8\" (UID: \"6e7da002-bfa6-4527-bfb9-ee3bc36677f1\") " pod="openshift-multus/multus-clbg8" Jul 09 09:33:35.344757 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.343322 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1677420d-7376-45ad-ac67-20128aec5756-var-lib-openvswitch\") pod \"ovnkube-node-xf4sw\" (UID: \"1677420d-7376-45ad-ac67-20128aec5756\") " pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" Jul 09 09:33:35.344757 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.343330 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/46a45a21-2ddc-4b0b-9988-911cde1598f7-var-lib-kubelet\") pod \"tuned-dnzdl\" (UID: \"46a45a21-2ddc-4b0b-9988-911cde1598f7\") " pod="openshift-cluster-node-tuning-operator/tuned-dnzdl" Jul 09 09:33:35.344757 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.343372 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/65d9652f-67de-405c-8e07-aa737e9205d8-system-cni-dir\") pod \"multus-additional-cni-plugins-rspgp\" (UID: \"65d9652f-67de-405c-8e07-aa737e9205d8\") " pod="openshift-multus/multus-additional-cni-plugins-rspgp" Jul 09 09:33:35.344757 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.343414 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/65d9652f-67de-405c-8e07-aa737e9205d8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rspgp\" (UID: \"65d9652f-67de-405c-8e07-aa737e9205d8\") " pod="openshift-multus/multus-additional-cni-plugins-rspgp" Jul 09 09:33:35.344757 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.343433 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6e7da002-bfa6-4527-bfb9-ee3bc36677f1-etc-kubernetes\") pod \"multus-clbg8\" (UID: \"6e7da002-bfa6-4527-bfb9-ee3bc36677f1\") " pod="openshift-multus/multus-clbg8" Jul 09 09:33:35.344757 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.343469 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1677420d-7376-45ad-ac67-20128aec5756-host-slash\") pod \"ovnkube-node-xf4sw\" (UID: \"1677420d-7376-45ad-ac67-20128aec5756\") " pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" Jul 09 09:33:35.344757 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.343493 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1677420d-7376-45ad-ac67-20128aec5756-host-cni-netd\") pod \"ovnkube-node-xf4sw\" (UID: \"1677420d-7376-45ad-ac67-20128aec5756\") " pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" Jul 09 09:33:35.344757 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.343503 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6e7da002-bfa6-4527-bfb9-ee3bc36677f1-etc-kubernetes\") pod \"multus-clbg8\" (UID: \"6e7da002-bfa6-4527-bfb9-ee3bc36677f1\") " pod="openshift-multus/multus-clbg8" Jul 09 09:33:35.344757 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.343509 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jpg2\" (UniqueName: \"kubernetes.io/projected/c0652b01-6ddc-4820-8fbe-c2e5b643f663-kube-api-access-6jpg2\") pod \"network-metrics-daemon-wsdxs\" (UID: \"c0652b01-6ddc-4820-8fbe-c2e5b643f663\") " pod="openshift-multus/network-metrics-daemon-wsdxs" Jul 09 09:33:35.344757 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.343541 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/46a45a21-2ddc-4b0b-9988-911cde1598f7-etc-modprobe-d\") pod \"tuned-dnzdl\" (UID: \"46a45a21-2ddc-4b0b-9988-911cde1598f7\") " pod="openshift-cluster-node-tuning-operator/tuned-dnzdl" Jul 09 09:33:35.344757 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.343550 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1677420d-7376-45ad-ac67-20128aec5756-host-slash\") pod \"ovnkube-node-xf4sw\" (UID: \"1677420d-7376-45ad-ac67-20128aec5756\") " pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" Jul 09 09:33:35.344757 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.343558 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1677420d-7376-45ad-ac67-20128aec5756-host-cni-netd\") pod \"ovnkube-node-xf4sw\" (UID: \"1677420d-7376-45ad-ac67-20128aec5756\") " pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" Jul 09 09:33:35.345250 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.343566 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qczpx\" (UniqueName: \"kubernetes.io/projected/65d9652f-67de-405c-8e07-aa737e9205d8-kube-api-access-qczpx\") pod \"multus-additional-cni-plugins-rspgp\" (UID: \"65d9652f-67de-405c-8e07-aa737e9205d8\") " pod="openshift-multus/multus-additional-cni-plugins-rspgp" Jul 09 09:33:35.345250 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.343598 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f2d0ed74-687b-403d-9ad5-17ee9fc39d6a-registration-dir\") pod \"aws-ebs-csi-driver-node-z76mc\" (UID: \"f2d0ed74-687b-403d-9ad5-17ee9fc39d6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z76mc" Jul 09 09:33:35.345250 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.343644 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f2d0ed74-687b-403d-9ad5-17ee9fc39d6a-registration-dir\") pod \"aws-ebs-csi-driver-node-z76mc\" (UID: \"f2d0ed74-687b-403d-9ad5-17ee9fc39d6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z76mc" Jul 09 09:33:35.345250 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.343657 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1677420d-7376-45ad-ac67-20128aec5756-node-log\") pod \"ovnkube-node-xf4sw\" (UID: \"1677420d-7376-45ad-ac67-20128aec5756\") " pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" Jul 09 09:33:35.345250 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.343674 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1677420d-7376-45ad-ac67-20128aec5756-host-cni-bin\") pod \"ovnkube-node-xf4sw\" (UID: \"1677420d-7376-45ad-ac67-20128aec5756\") " pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" Jul 09 09:33:35.345250 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.343682 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1677420d-7376-45ad-ac67-20128aec5756-node-log\") pod \"ovnkube-node-xf4sw\" (UID: \"1677420d-7376-45ad-ac67-20128aec5756\") " pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" Jul 09 09:33:35.345250 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.343691 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5rnf\" (UniqueName: \"kubernetes.io/projected/1677420d-7376-45ad-ac67-20128aec5756-kube-api-access-c5rnf\") pod \"ovnkube-node-xf4sw\" (UID: \"1677420d-7376-45ad-ac67-20128aec5756\") " pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" Jul 09 09:33:35.345250 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.343703 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6e7da002-bfa6-4527-bfb9-ee3bc36677f1-cni-binary-copy\") pod \"multus-clbg8\" (UID: \"6e7da002-bfa6-4527-bfb9-ee3bc36677f1\") " pod="openshift-multus/multus-clbg8" Jul 09 09:33:35.345250 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.343716 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48bc9\" (UniqueName: \"kubernetes.io/projected/53e2e8a5-34d5-4317-9733-143a53880b9b-kube-api-access-48bc9\") pod \"network-check-target-gqlhc\" (UID: \"53e2e8a5-34d5-4317-9733-143a53880b9b\") " pod="openshift-network-diagnostics/network-check-target-gqlhc" Jul 09 09:33:35.345250 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.343718 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1677420d-7376-45ad-ac67-20128aec5756-host-cni-bin\") pod \"ovnkube-node-xf4sw\" (UID: \"1677420d-7376-45ad-ac67-20128aec5756\") " pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" Jul 09 09:33:35.345250 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.343748 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/46a45a21-2ddc-4b0b-9988-911cde1598f7-run\") pod \"tuned-dnzdl\" (UID: \"46a45a21-2ddc-4b0b-9988-911cde1598f7\") " pod="openshift-cluster-node-tuning-operator/tuned-dnzdl" Jul 09 09:33:35.345250 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.343736 2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 09 09:33:35.345250 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.343767 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/65d9652f-67de-405c-8e07-aa737e9205d8-os-release\") pod \"multus-additional-cni-plugins-rspgp\" (UID: \"65d9652f-67de-405c-8e07-aa737e9205d8\") " pod="openshift-multus/multus-additional-cni-plugins-rspgp" Jul 09 09:33:35.345250 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.343786 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1677420d-7376-45ad-ac67-20128aec5756-host-run-netns\") pod \"ovnkube-node-xf4sw\" (UID: \"1677420d-7376-45ad-ac67-20128aec5756\") " pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" Jul 09 09:33:35.345250 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.343809 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1677420d-7376-45ad-ac67-20128aec5756-run-ovn\") pod \"ovnkube-node-xf4sw\" (UID: \"1677420d-7376-45ad-ac67-20128aec5756\") " pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" Jul 09 09:33:35.345250 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.343827 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1677420d-7376-45ad-ac67-20128aec5756-ovnkube-script-lib\") pod \"ovnkube-node-xf4sw\" (UID: \"1677420d-7376-45ad-ac67-20128aec5756\") " pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" Jul 09 09:33:35.345250 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.343835 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1677420d-7376-45ad-ac67-20128aec5756-host-run-netns\") pod \"ovnkube-node-xf4sw\" (UID: \"1677420d-7376-45ad-ac67-20128aec5756\") " pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" Jul 09 09:33:35.345721 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.343847 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f8587efa-7f0b-4a11-b03f-e08b942aa1f1-host\") pod \"node-ca-qbtkh\" (UID: \"f8587efa-7f0b-4a11-b03f-e08b942aa1f1\") " pod="openshift-image-registry/node-ca-qbtkh" Jul 09 09:33:35.345721 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.343859 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1677420d-7376-45ad-ac67-20128aec5756-env-overrides\") pod \"ovnkube-node-xf4sw\" (UID: \"1677420d-7376-45ad-ac67-20128aec5756\") " pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" Jul 09 09:33:35.345721 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.343877 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1677420d-7376-45ad-ac67-20128aec5756-run-ovn\") pod \"ovnkube-node-xf4sw\" (UID: \"1677420d-7376-45ad-ac67-20128aec5756\") " pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" Jul 09 09:33:35.345721 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.343999 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1677420d-7376-45ad-ac67-20128aec5756-ovnkube-config\") pod \"ovnkube-node-xf4sw\" (UID: \"1677420d-7376-45ad-ac67-20128aec5756\") " pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" Jul 09 09:33:35.345721 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.344197 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1677420d-7376-45ad-ac67-20128aec5756-ovnkube-script-lib\") pod \"ovnkube-node-xf4sw\" (UID: \"1677420d-7376-45ad-ac67-20128aec5756\") " pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" Jul 09 09:33:35.346510 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.346496 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1677420d-7376-45ad-ac67-20128aec5756-ovn-node-metrics-cert\") pod \"ovnkube-node-xf4sw\" (UID: \"1677420d-7376-45ad-ac67-20128aec5756\") " pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" Jul 09 09:33:35.363136 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.363108 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z56tr\" (UniqueName: \"kubernetes.io/projected/c7314861-eada-458d-bf61-21da88f8b1c8-kube-api-access-z56tr\") pod \"iptables-alerter-r6npq\" (UID: \"c7314861-eada-458d-bf61-21da88f8b1c8\") " pod="openshift-network-operator/iptables-alerter-r6npq" Jul 09 09:33:35.363210 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.363144 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvbxg\" (UniqueName: \"kubernetes.io/projected/6e7da002-bfa6-4527-bfb9-ee3bc36677f1-kube-api-access-vvbxg\") pod \"multus-clbg8\" (UID: \"6e7da002-bfa6-4527-bfb9-ee3bc36677f1\") " pod="openshift-multus/multus-clbg8" Jul 09 09:33:35.365330 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.365310 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5rnf\" (UniqueName: \"kubernetes.io/projected/1677420d-7376-45ad-ac67-20128aec5756-kube-api-access-c5rnf\") pod \"ovnkube-node-xf4sw\" (UID: \"1677420d-7376-45ad-ac67-20128aec5756\") " pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" Jul 09 09:33:35.365405 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.365356 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n8vt\" (UniqueName: \"kubernetes.io/projected/f2d0ed74-687b-403d-9ad5-17ee9fc39d6a-kube-api-access-5n8vt\") pod \"aws-ebs-csi-driver-node-z76mc\" (UID: \"f2d0ed74-687b-403d-9ad5-17ee9fc39d6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z76mc" Jul 09 09:33:35.444120 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.444087 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlvnh\" (UniqueName: \"kubernetes.io/projected/f8587efa-7f0b-4a11-b03f-e08b942aa1f1-kube-api-access-dlvnh\") pod \"node-ca-qbtkh\" (UID: \"f8587efa-7f0b-4a11-b03f-e08b942aa1f1\") " pod="openshift-image-registry/node-ca-qbtkh" Jul 09 09:33:35.444120 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.444119 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/32307b23-721f-4ed0-add4-542ed80a29d8-hosts-file\") pod \"node-resolver-nx4l4\" (UID: \"32307b23-721f-4ed0-add4-542ed80a29d8\") " pod="openshift-dns/node-resolver-nx4l4" Jul 09 09:33:35.444348 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.444136 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a0c24dc3-5e21-4537-966b-0c5f9e46f8d9-kubelet-config\") pod \"global-pull-secret-syncer-sw9zq\" (UID: \"a0c24dc3-5e21-4537-966b-0c5f9e46f8d9\") " pod="kube-system/global-pull-secret-syncer-sw9zq" Jul 09 09:33:35.444348 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.444153 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dc248a4e-4aa6-4ea4-81b7-439e669b56c1-konnectivity-ca\") pod \"konnectivity-agent-gbmbr\" (UID: \"dc248a4e-4aa6-4ea4-81b7-439e669b56c1\") " pod="kube-system/konnectivity-agent-gbmbr" Jul 09 09:33:35.444348 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.444173 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/46a45a21-2ddc-4b0b-9988-911cde1598f7-etc-sysconfig\") pod \"tuned-dnzdl\" (UID: \"46a45a21-2ddc-4b0b-9988-911cde1598f7\") " pod="openshift-cluster-node-tuning-operator/tuned-dnzdl" Jul 09 09:33:35.444348 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.444195 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/65d9652f-67de-405c-8e07-aa737e9205d8-cni-binary-copy\") pod \"multus-additional-cni-plugins-rspgp\" (UID: \"65d9652f-67de-405c-8e07-aa737e9205d8\") " pod="openshift-multus/multus-additional-cni-plugins-rspgp" Jul 09 09:33:35.444348 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.444217 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/46a45a21-2ddc-4b0b-9988-911cde1598f7-etc-sysctl-conf\") pod \"tuned-dnzdl\" (UID: \"46a45a21-2ddc-4b0b-9988-911cde1598f7\") " pod="openshift-cluster-node-tuning-operator/tuned-dnzdl" Jul 09 09:33:35.444348 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.444223 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/32307b23-721f-4ed0-add4-542ed80a29d8-hosts-file\") pod \"node-resolver-nx4l4\" (UID: \"32307b23-721f-4ed0-add4-542ed80a29d8\") " pod="openshift-dns/node-resolver-nx4l4" Jul 09 09:33:35.444348 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.444244 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/46a45a21-2ddc-4b0b-9988-911cde1598f7-etc-tuned\") pod \"tuned-dnzdl\" (UID: \"46a45a21-2ddc-4b0b-9988-911cde1598f7\") " pod="openshift-cluster-node-tuning-operator/tuned-dnzdl" Jul 09 09:33:35.444348 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.444267 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0652b01-6ddc-4820-8fbe-c2e5b643f663-metrics-certs\") pod \"network-metrics-daemon-wsdxs\" (UID: \"c0652b01-6ddc-4820-8fbe-c2e5b643f663\") " pod="openshift-multus/network-metrics-daemon-wsdxs" Jul 09 09:33:35.444348 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.444264 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/46a45a21-2ddc-4b0b-9988-911cde1598f7-etc-sysconfig\") pod \"tuned-dnzdl\" (UID: \"46a45a21-2ddc-4b0b-9988-911cde1598f7\") " pod="openshift-cluster-node-tuning-operator/tuned-dnzdl" Jul 09 09:33:35.444348 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.444266 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a0c24dc3-5e21-4537-966b-0c5f9e46f8d9-kubelet-config\") pod \"global-pull-secret-syncer-sw9zq\" (UID: \"a0c24dc3-5e21-4537-966b-0c5f9e46f8d9\") " pod="kube-system/global-pull-secret-syncer-sw9zq" Jul 09 09:33:35.444348 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.444314 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/65d9652f-67de-405c-8e07-aa737e9205d8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rspgp\" (UID: \"65d9652f-67de-405c-8e07-aa737e9205d8\") " pod="openshift-multus/multus-additional-cni-plugins-rspgp" Jul 09 09:33:35.444348 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.444346 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dc248a4e-4aa6-4ea4-81b7-439e669b56c1-agent-certs\") pod \"konnectivity-agent-gbmbr\" (UID: \"dc248a4e-4aa6-4ea4-81b7-439e669b56c1\") " pod="kube-system/konnectivity-agent-gbmbr" Jul 09 09:33:35.444940 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.444372 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f8587efa-7f0b-4a11-b03f-e08b942aa1f1-serviceca\") pod \"node-ca-qbtkh\" (UID: \"f8587efa-7f0b-4a11-b03f-e08b942aa1f1\") " pod="openshift-image-registry/node-ca-qbtkh" Jul 09 09:33:35.444940 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.444359 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/46a45a21-2ddc-4b0b-9988-911cde1598f7-etc-sysctl-conf\") pod \"tuned-dnzdl\" (UID: \"46a45a21-2ddc-4b0b-9988-911cde1598f7\") " pod="openshift-cluster-node-tuning-operator/tuned-dnzdl" Jul 09 09:33:35.444940 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.444400 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/46a45a21-2ddc-4b0b-9988-911cde1598f7-etc-sysctl-d\") pod \"tuned-dnzdl\" (UID: \"46a45a21-2ddc-4b0b-9988-911cde1598f7\") " pod="openshift-cluster-node-tuning-operator/tuned-dnzdl" Jul 09 09:33:35.444940 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.444423 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/46a45a21-2ddc-4b0b-9988-911cde1598f7-lib-modules\") pod \"tuned-dnzdl\" (UID: \"46a45a21-2ddc-4b0b-9988-911cde1598f7\") " pod="openshift-cluster-node-tuning-operator/tuned-dnzdl" Jul 09 09:33:35.444940 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:35.444428 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 09:33:35.444940 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.444463 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/46a45a21-2ddc-4b0b-9988-911cde1598f7-host\") pod \"tuned-dnzdl\" (UID: \"46a45a21-2ddc-4b0b-9988-911cde1598f7\") " pod="openshift-cluster-node-tuning-operator/tuned-dnzdl" Jul 09 09:33:35.444940 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.444485 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/46a45a21-2ddc-4b0b-9988-911cde1598f7-tmp\") pod \"tuned-dnzdl\" (UID: \"46a45a21-2ddc-4b0b-9988-911cde1598f7\") " pod="openshift-cluster-node-tuning-operator/tuned-dnzdl" Jul 09 09:33:35.444940 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.444514 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32307b23-721f-4ed0-add4-542ed80a29d8-tmp-dir\") pod \"node-resolver-nx4l4\" (UID: \"32307b23-721f-4ed0-add4-542ed80a29d8\") " pod="openshift-dns/node-resolver-nx4l4" Jul 09 09:33:35.444940 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.444540 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87rbm\" (UniqueName: \"kubernetes.io/projected/46a45a21-2ddc-4b0b-9988-911cde1598f7-kube-api-access-87rbm\") pod \"tuned-dnzdl\" (UID: \"46a45a21-2ddc-4b0b-9988-911cde1598f7\") " pod="openshift-cluster-node-tuning-operator/tuned-dnzdl" Jul 09 09:33:35.444940 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:35.444560 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0652b01-6ddc-4820-8fbe-c2e5b643f663-metrics-certs podName:c0652b01-6ddc-4820-8fbe-c2e5b643f663 nodeName:}" failed. No retries permitted until 2026-07-09 09:33:35.944536088 +0000 UTC m=+2.095691233 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0652b01-6ddc-4820-8fbe-c2e5b643f663-metrics-certs") pod "network-metrics-daemon-wsdxs" (UID: "c0652b01-6ddc-4820-8fbe-c2e5b643f663") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 09:33:35.444940 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.444594 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/65d9652f-67de-405c-8e07-aa737e9205d8-cnibin\") pod \"multus-additional-cni-plugins-rspgp\" (UID: \"65d9652f-67de-405c-8e07-aa737e9205d8\") " pod="openshift-multus/multus-additional-cni-plugins-rspgp" Jul 09 09:33:35.444940 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.444627 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a0c24dc3-5e21-4537-966b-0c5f9e46f8d9-original-pull-secret\") pod \"global-pull-secret-syncer-sw9zq\" (UID: \"a0c24dc3-5e21-4537-966b-0c5f9e46f8d9\") " pod="kube-system/global-pull-secret-syncer-sw9zq" Jul 09 09:33:35.444940 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.444656 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a0c24dc3-5e21-4537-966b-0c5f9e46f8d9-dbus\") pod \"global-pull-secret-syncer-sw9zq\" (UID: \"a0c24dc3-5e21-4537-966b-0c5f9e46f8d9\") " pod="kube-system/global-pull-secret-syncer-sw9zq" Jul 09 09:33:35.444940 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.444682 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/46a45a21-2ddc-4b0b-9988-911cde1598f7-etc-systemd\") pod \"tuned-dnzdl\" (UID: \"46a45a21-2ddc-4b0b-9988-911cde1598f7\") " pod="openshift-cluster-node-tuning-operator/tuned-dnzdl" Jul 09 09:33:35.444940 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.444703 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/46a45a21-2ddc-4b0b-9988-911cde1598f7-sys\") pod \"tuned-dnzdl\" (UID: \"46a45a21-2ddc-4b0b-9988-911cde1598f7\") " pod="openshift-cluster-node-tuning-operator/tuned-dnzdl" Jul 09 09:33:35.444940 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.444727 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/65d9652f-67de-405c-8e07-aa737e9205d8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rspgp\" (UID: \"65d9652f-67de-405c-8e07-aa737e9205d8\") " pod="openshift-multus/multus-additional-cni-plugins-rspgp" Jul 09 09:33:35.444940 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.444755 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/46a45a21-2ddc-4b0b-9988-911cde1598f7-var-lib-kubelet\") pod \"tuned-dnzdl\" (UID: \"46a45a21-2ddc-4b0b-9988-911cde1598f7\") " pod="openshift-cluster-node-tuning-operator/tuned-dnzdl" Jul 09 09:33:35.445795 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.444777 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/65d9652f-67de-405c-8e07-aa737e9205d8-system-cni-dir\") pod \"multus-additional-cni-plugins-rspgp\" (UID: \"65d9652f-67de-405c-8e07-aa737e9205d8\") " pod="openshift-multus/multus-additional-cni-plugins-rspgp" Jul 09 09:33:35.445795 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.444777 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dc248a4e-4aa6-4ea4-81b7-439e669b56c1-konnectivity-ca\") pod \"konnectivity-agent-gbmbr\" (UID: \"dc248a4e-4aa6-4ea4-81b7-439e669b56c1\") " pod="kube-system/konnectivity-agent-gbmbr" Jul 09 09:33:35.445795 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.444815 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/65d9652f-67de-405c-8e07-aa737e9205d8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rspgp\" (UID: \"65d9652f-67de-405c-8e07-aa737e9205d8\") " pod="openshift-multus/multus-additional-cni-plugins-rspgp" Jul 09 09:33:35.445795 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.444821 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/65d9652f-67de-405c-8e07-aa737e9205d8-system-cni-dir\") pod \"multus-additional-cni-plugins-rspgp\" (UID: \"65d9652f-67de-405c-8e07-aa737e9205d8\") " pod="openshift-multus/multus-additional-cni-plugins-rspgp" Jul 09 09:33:35.445795 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.444837 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/65d9652f-67de-405c-8e07-aa737e9205d8-cni-binary-copy\") pod \"multus-additional-cni-plugins-rspgp\" (UID: \"65d9652f-67de-405c-8e07-aa737e9205d8\") " pod="openshift-multus/multus-additional-cni-plugins-rspgp" Jul 09 09:33:35.445795 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.444987 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f8587efa-7f0b-4a11-b03f-e08b942aa1f1-serviceca\") pod \"node-ca-qbtkh\" (UID: \"f8587efa-7f0b-4a11-b03f-e08b942aa1f1\") " pod="openshift-image-registry/node-ca-qbtkh" Jul 09 09:33:35.445795 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.444846 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jpg2\" (UniqueName: \"kubernetes.io/projected/c0652b01-6ddc-4820-8fbe-c2e5b643f663-kube-api-access-6jpg2\") pod \"network-metrics-daemon-wsdxs\" (UID: \"c0652b01-6ddc-4820-8fbe-c2e5b643f663\") " pod="openshift-multus/network-metrics-daemon-wsdxs" Jul 09 09:33:35.445795 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.445084 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/46a45a21-2ddc-4b0b-9988-911cde1598f7-etc-modprobe-d\") pod \"tuned-dnzdl\" (UID: \"46a45a21-2ddc-4b0b-9988-911cde1598f7\") " pod="openshift-cluster-node-tuning-operator/tuned-dnzdl" Jul 09 09:33:35.445795 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.445097 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/46a45a21-2ddc-4b0b-9988-911cde1598f7-etc-sysctl-d\") pod \"tuned-dnzdl\" (UID: \"46a45a21-2ddc-4b0b-9988-911cde1598f7\") " pod="openshift-cluster-node-tuning-operator/tuned-dnzdl" Jul 09 09:33:35.445795 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.445118 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qczpx\" (UniqueName: \"kubernetes.io/projected/65d9652f-67de-405c-8e07-aa737e9205d8-kube-api-access-qczpx\") pod \"multus-additional-cni-plugins-rspgp\" (UID: \"65d9652f-67de-405c-8e07-aa737e9205d8\") " pod="openshift-multus/multus-additional-cni-plugins-rspgp" Jul 09 09:33:35.445795 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.445156 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48bc9\" (UniqueName: \"kubernetes.io/projected/53e2e8a5-34d5-4317-9733-143a53880b9b-kube-api-access-48bc9\") pod \"network-check-target-gqlhc\" (UID: \"53e2e8a5-34d5-4317-9733-143a53880b9b\") " pod="openshift-network-diagnostics/network-check-target-gqlhc" Jul 09 09:33:35.445795 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.445180 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/46a45a21-2ddc-4b0b-9988-911cde1598f7-lib-modules\") pod \"tuned-dnzdl\" (UID: \"46a45a21-2ddc-4b0b-9988-911cde1598f7\") " pod="openshift-cluster-node-tuning-operator/tuned-dnzdl" Jul 09 09:33:35.445795 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.445181 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/46a45a21-2ddc-4b0b-9988-911cde1598f7-run\") pod \"tuned-dnzdl\" (UID: \"46a45a21-2ddc-4b0b-9988-911cde1598f7\") " pod="openshift-cluster-node-tuning-operator/tuned-dnzdl" Jul 09 09:33:35.445795 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.445214 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a0c24dc3-5e21-4537-966b-0c5f9e46f8d9-dbus\") pod \"global-pull-secret-syncer-sw9zq\" (UID: \"a0c24dc3-5e21-4537-966b-0c5f9e46f8d9\") " pod="kube-system/global-pull-secret-syncer-sw9zq" Jul 09 09:33:35.445795 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.445223 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/46a45a21-2ddc-4b0b-9988-911cde1598f7-run\") pod \"tuned-dnzdl\" (UID: \"46a45a21-2ddc-4b0b-9988-911cde1598f7\") " pod="openshift-cluster-node-tuning-operator/tuned-dnzdl" Jul 09 09:33:35.445795 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.445215 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/65d9652f-67de-405c-8e07-aa737e9205d8-os-release\") pod \"multus-additional-cni-plugins-rspgp\" (UID: \"65d9652f-67de-405c-8e07-aa737e9205d8\") " pod="openshift-multus/multus-additional-cni-plugins-rspgp" Jul 09 09:33:35.445795 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.445260 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f8587efa-7f0b-4a11-b03f-e08b942aa1f1-host\") pod \"node-ca-qbtkh\" (UID: \"f8587efa-7f0b-4a11-b03f-e08b942aa1f1\") " pod="openshift-image-registry/node-ca-qbtkh" Jul 09 09:33:35.446838 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.445268 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/46a45a21-2ddc-4b0b-9988-911cde1598f7-etc-systemd\") pod \"tuned-dnzdl\" (UID: \"46a45a21-2ddc-4b0b-9988-911cde1598f7\") " pod="openshift-cluster-node-tuning-operator/tuned-dnzdl" Jul 09 09:33:35.446838 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.445284 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqtj4\" (UniqueName: \"kubernetes.io/projected/32307b23-721f-4ed0-add4-542ed80a29d8-kube-api-access-bqtj4\") pod \"node-resolver-nx4l4\" (UID: \"32307b23-721f-4ed0-add4-542ed80a29d8\") " pod="openshift-dns/node-resolver-nx4l4" Jul 09 09:33:35.446838 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.445293 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/46a45a21-2ddc-4b0b-9988-911cde1598f7-host\") pod \"tuned-dnzdl\" (UID: \"46a45a21-2ddc-4b0b-9988-911cde1598f7\") " pod="openshift-cluster-node-tuning-operator/tuned-dnzdl" Jul 09 09:33:35.446838 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.445306 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46a45a21-2ddc-4b0b-9988-911cde1598f7-etc-kubernetes\") pod \"tuned-dnzdl\" (UID: \"46a45a21-2ddc-4b0b-9988-911cde1598f7\") " pod="openshift-cluster-node-tuning-operator/tuned-dnzdl" Jul 09 09:33:35.446838 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.445312 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/46a45a21-2ddc-4b0b-9988-911cde1598f7-sys\") pod \"tuned-dnzdl\" (UID: \"46a45a21-2ddc-4b0b-9988-911cde1598f7\") " pod="openshift-cluster-node-tuning-operator/tuned-dnzdl" Jul 09 09:33:35.446838 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.445314 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/65d9652f-67de-405c-8e07-aa737e9205d8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rspgp\" (UID: \"65d9652f-67de-405c-8e07-aa737e9205d8\") " pod="openshift-multus/multus-additional-cni-plugins-rspgp" Jul 09 09:33:35.446838 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.445321 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/65d9652f-67de-405c-8e07-aa737e9205d8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rspgp\" (UID: \"65d9652f-67de-405c-8e07-aa737e9205d8\") " pod="openshift-multus/multus-additional-cni-plugins-rspgp" Jul 09 09:33:35.446838 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.445261 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/65d9652f-67de-405c-8e07-aa737e9205d8-os-release\") pod \"multus-additional-cni-plugins-rspgp\" (UID: \"65d9652f-67de-405c-8e07-aa737e9205d8\") " pod="openshift-multus/multus-additional-cni-plugins-rspgp" Jul 09 09:33:35.446838 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.445384 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f8587efa-7f0b-4a11-b03f-e08b942aa1f1-host\") pod \"node-ca-qbtkh\" (UID: \"f8587efa-7f0b-4a11-b03f-e08b942aa1f1\") " pod="openshift-image-registry/node-ca-qbtkh" Jul 09 09:33:35.446838 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.445157 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/65d9652f-67de-405c-8e07-aa737e9205d8-cnibin\") pod \"multus-additional-cni-plugins-rspgp\" (UID: \"65d9652f-67de-405c-8e07-aa737e9205d8\") " pod="openshift-multus/multus-additional-cni-plugins-rspgp" Jul 09 09:33:35.446838 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:35.445386 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 09:33:35.446838 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.445385 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46a45a21-2ddc-4b0b-9988-911cde1598f7-etc-kubernetes\") pod \"tuned-dnzdl\" (UID: \"46a45a21-2ddc-4b0b-9988-911cde1598f7\") " pod="openshift-cluster-node-tuning-operator/tuned-dnzdl" Jul 09 09:33:35.446838 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.445476 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/65d9652f-67de-405c-8e07-aa737e9205d8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rspgp\" (UID: \"65d9652f-67de-405c-8e07-aa737e9205d8\") " pod="openshift-multus/multus-additional-cni-plugins-rspgp" Jul 09 09:33:35.446838 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:35.445540 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0c24dc3-5e21-4537-966b-0c5f9e46f8d9-original-pull-secret podName:a0c24dc3-5e21-4537-966b-0c5f9e46f8d9 nodeName:}" failed. No retries permitted until 2026-07-09 09:33:35.945523849 +0000 UTC m=+2.096678970 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a0c24dc3-5e21-4537-966b-0c5f9e46f8d9-original-pull-secret") pod "global-pull-secret-syncer-sw9zq" (UID: "a0c24dc3-5e21-4537-966b-0c5f9e46f8d9") : object "kube-system"/"original-pull-secret" not registered Jul 09 09:33:35.446838 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.445556 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/46a45a21-2ddc-4b0b-9988-911cde1598f7-var-lib-kubelet\") pod \"tuned-dnzdl\" (UID: \"46a45a21-2ddc-4b0b-9988-911cde1598f7\") " pod="openshift-cluster-node-tuning-operator/tuned-dnzdl" Jul 09 09:33:35.446838 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.445723 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/46a45a21-2ddc-4b0b-9988-911cde1598f7-etc-modprobe-d\") pod \"tuned-dnzdl\" (UID: \"46a45a21-2ddc-4b0b-9988-911cde1598f7\") " pod="openshift-cluster-node-tuning-operator/tuned-dnzdl" Jul 09 09:33:35.446838 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.445791 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32307b23-721f-4ed0-add4-542ed80a29d8-tmp-dir\") pod \"node-resolver-nx4l4\" (UID: \"32307b23-721f-4ed0-add4-542ed80a29d8\") " pod="openshift-dns/node-resolver-nx4l4" Jul 09 09:33:35.447745 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.446932 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dc248a4e-4aa6-4ea4-81b7-439e669b56c1-agent-certs\") pod \"konnectivity-agent-gbmbr\" (UID: \"dc248a4e-4aa6-4ea4-81b7-439e669b56c1\") " pod="kube-system/konnectivity-agent-gbmbr" Jul 09 09:33:35.448957 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.448937 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/46a45a21-2ddc-4b0b-9988-911cde1598f7-etc-tuned\") pod \"tuned-dnzdl\" (UID: \"46a45a21-2ddc-4b0b-9988-911cde1598f7\") " pod="openshift-cluster-node-tuning-operator/tuned-dnzdl" Jul 09 09:33:35.449049 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.448981 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/46a45a21-2ddc-4b0b-9988-911cde1598f7-tmp\") pod \"tuned-dnzdl\" (UID: \"46a45a21-2ddc-4b0b-9988-911cde1598f7\") " pod="openshift-cluster-node-tuning-operator/tuned-dnzdl" Jul 09 09:33:35.457913 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:35.457896 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 09:33:35.457913 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:35.457915 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 09:33:35.458030 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:35.457925 2578 projected.go:196] Error preparing data for projected volume kube-api-access-48bc9 for pod openshift-network-diagnostics/network-check-target-gqlhc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 09:33:35.458030 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:35.457974 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/53e2e8a5-34d5-4317-9733-143a53880b9b-kube-api-access-48bc9 podName:53e2e8a5-34d5-4317-9733-143a53880b9b nodeName:}" failed. No retries permitted until 2026-07-09 09:33:35.957962054 +0000 UTC m=+2.109117170 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-48bc9" (UniqueName: "kubernetes.io/projected/53e2e8a5-34d5-4317-9733-143a53880b9b-kube-api-access-48bc9") pod "network-check-target-gqlhc" (UID: "53e2e8a5-34d5-4317-9733-143a53880b9b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 09:33:35.461187 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.461161 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqtj4\" (UniqueName: \"kubernetes.io/projected/32307b23-721f-4ed0-add4-542ed80a29d8-kube-api-access-bqtj4\") pod \"node-resolver-nx4l4\" (UID: \"32307b23-721f-4ed0-add4-542ed80a29d8\") " pod="openshift-dns/node-resolver-nx4l4" Jul 09 09:33:35.461643 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.461624 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jpg2\" (UniqueName: \"kubernetes.io/projected/c0652b01-6ddc-4820-8fbe-c2e5b643f663-kube-api-access-6jpg2\") pod \"network-metrics-daemon-wsdxs\" (UID: \"c0652b01-6ddc-4820-8fbe-c2e5b643f663\") " pod="openshift-multus/network-metrics-daemon-wsdxs" Jul 09 09:33:35.461810 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.461793 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87rbm\" (UniqueName: \"kubernetes.io/projected/46a45a21-2ddc-4b0b-9988-911cde1598f7-kube-api-access-87rbm\") pod \"tuned-dnzdl\" (UID: \"46a45a21-2ddc-4b0b-9988-911cde1598f7\") " pod="openshift-cluster-node-tuning-operator/tuned-dnzdl" Jul 09 09:33:35.462170 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.462151 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qczpx\" (UniqueName: \"kubernetes.io/projected/65d9652f-67de-405c-8e07-aa737e9205d8-kube-api-access-qczpx\") pod \"multus-additional-cni-plugins-rspgp\" (UID: \"65d9652f-67de-405c-8e07-aa737e9205d8\") " pod="openshift-multus/multus-additional-cni-plugins-rspgp" Jul 09 09:33:35.462390 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.462375 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlvnh\" (UniqueName: \"kubernetes.io/projected/f8587efa-7f0b-4a11-b03f-e08b942aa1f1-kube-api-access-dlvnh\") pod \"node-ca-qbtkh\" (UID: \"f8587efa-7f0b-4a11-b03f-e08b942aa1f1\") " pod="openshift-image-registry/node-ca-qbtkh" Jul 09 09:33:35.513510 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:35.513480 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8f995f6fcf6abd8e78373eeaaaf3c86.slice/crio-0616572aa16433c7e591c2b0e019281b5f392607c0927a601c9fe1beb38dc3d7 WatchSource:0}: Error finding container 0616572aa16433c7e591c2b0e019281b5f392607c0927a601c9fe1beb38dc3d7: Status 404 returned error can't find the container with id 0616572aa16433c7e591c2b0e019281b5f392607c0927a601c9fe1beb38dc3d7 Jul 09 09:33:35.514033 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:35.514010 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4386692e678e561186646b99422b104a.slice/crio-b834a74edb8216d9b464c79b1607c338b4cf53e304f2a98d66551b80e1f4350c WatchSource:0}: Error finding container b834a74edb8216d9b464c79b1607c338b4cf53e304f2a98d66551b80e1f4350c: Status 404 returned error can't find the container with id b834a74edb8216d9b464c79b1607c338b4cf53e304f2a98d66551b80e1f4350c Jul 09 09:33:35.519217 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.519203 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 09:33:35.566614 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.566567 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z76mc" Jul 09 09:33:35.574074 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:35.574056 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2d0ed74_687b_403d_9ad5_17ee9fc39d6a.slice/crio-72e52dadbdb74f343eeb3128e3943b6d16c4619ad4e735f7e1df7c71428e8e7e WatchSource:0}: Error finding container 72e52dadbdb74f343eeb3128e3943b6d16c4619ad4e735f7e1df7c71428e8e7e: Status 404 returned error can't find the container with id 72e52dadbdb74f343eeb3128e3943b6d16c4619ad4e735f7e1df7c71428e8e7e Jul 09 09:33:35.580502 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.580488 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-clbg8" Jul 09 09:33:35.587091 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:35.587069 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e7da002_bfa6_4527_bfb9_ee3bc36677f1.slice/crio-f418557b7904f348eba6a72b07619a6d204681be01c1e3960ac8cd575cfec868 WatchSource:0}: Error finding container f418557b7904f348eba6a72b07619a6d204681be01c1e3960ac8cd575cfec868: Status 404 returned error can't find the container with id f418557b7904f348eba6a72b07619a6d204681be01c1e3960ac8cd575cfec868 Jul 09 09:33:35.601226 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.601207 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-r6npq" Jul 09 09:33:35.608092 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:35.608075 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7314861_eada_458d_bf61_21da88f8b1c8.slice/crio-818669a0bae20786785de96265cf1630c7dc6a1f8fb46ade3cafd96587f647d5 WatchSource:0}: Error finding container 818669a0bae20786785de96265cf1630c7dc6a1f8fb46ade3cafd96587f647d5: Status 404 returned error can't find the container with id 818669a0bae20786785de96265cf1630c7dc6a1f8fb46ade3cafd96587f647d5 Jul 09 09:33:35.614829 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.614812 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" Jul 09 09:33:35.621550 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:35.621528 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1677420d_7376_45ad_ac67_20128aec5756.slice/crio-a97a533a92431cae133127b67f3bf585505a444d9556577013a55e569fe7ba3c WatchSource:0}: Error finding container a97a533a92431cae133127b67f3bf585505a444d9556577013a55e569fe7ba3c: Status 404 returned error can't find the container with id a97a533a92431cae133127b67f3bf585505a444d9556577013a55e569fe7ba3c Jul 09 09:33:35.637720 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.637704 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gbmbr" Jul 09 09:33:35.644041 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:35.644025 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc248a4e_4aa6_4ea4_81b7_439e669b56c1.slice/crio-04e3c9c361afc44356bbf961b5874bfce4558abb66268da97785c77a3dd743f0 WatchSource:0}: Error finding container 04e3c9c361afc44356bbf961b5874bfce4558abb66268da97785c77a3dd743f0: Status 404 returned error can't find the container with id 04e3c9c361afc44356bbf961b5874bfce4558abb66268da97785c77a3dd743f0 Jul 09 09:33:35.654242 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.654219 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dnzdl" Jul 09 09:33:35.660839 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:35.660821 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46a45a21_2ddc_4b0b_9988_911cde1598f7.slice/crio-4b8ee8a3df03802a8d0465facf2573cef977d911879757912af29cd570ce6af1 WatchSource:0}: Error finding container 4b8ee8a3df03802a8d0465facf2573cef977d911879757912af29cd570ce6af1: Status 404 returned error can't find the container with id 4b8ee8a3df03802a8d0465facf2573cef977d911879757912af29cd570ce6af1 Jul 09 09:33:35.675753 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.675738 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qbtkh" Jul 09 09:33:35.682328 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:35.682312 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8587efa_7f0b_4a11_b03f_e08b942aa1f1.slice/crio-95af8d0e71dc951ff96563c4b41ccff704f5bbb18f69b5ae668bcb25f201f263 WatchSource:0}: Error finding container 95af8d0e71dc951ff96563c4b41ccff704f5bbb18f69b5ae668bcb25f201f263: Status 404 returned error can't find the container with id 95af8d0e71dc951ff96563c4b41ccff704f5bbb18f69b5ae668bcb25f201f263 Jul 09 09:33:35.689432 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:35.689414 2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 09:33:36.756005 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:36.755990 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rspgp" Jul 09 09:33:36.760797 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:36.760778 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nx4l4" Jul 09 09:33:36.762477 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:36.762431 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65d9652f_67de_405c_8e07_aa737e9205d8.slice/crio-1c4fda396b263eb3748af6f2af99dc561fd01d701ca37e48dac906fcdf2657c4 WatchSource:0}: Error finding container 1c4fda396b263eb3748af6f2af99dc561fd01d701ca37e48dac906fcdf2657c4: Status 404 returned error can't find the container with id 1c4fda396b263eb3748af6f2af99dc561fd01d701ca37e48dac906fcdf2657c4 Jul 09 09:33:36.767887 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:33:36.767868 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32307b23_721f_4ed0_add4_542ed80a29d8.slice/crio-14ff7be5d35e573349118ccd1f988bb5d58354bde7fb1be4fb5f56cacd19d746 WatchSource:0}: Error finding container 14ff7be5d35e573349118ccd1f988bb5d58354bde7fb1be4fb5f56cacd19d746: Status 404 returned error can't find the container with id 14ff7be5d35e573349118ccd1f988bb5d58354bde7fb1be4fb5f56cacd19d746 Jul 09 09:33:36.769865 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:36.769514 2578 kuberuntime_manager.go:1459] "Unhandled Error" err="container dns-node-resolver start failed in pod node-resolver-nx4l4_openshift-dns(32307b23-721f-4ed0-add4-542ed80a29d8): ErrImagePull: pull QPS exceeded" logger="UnhandledError" Jul 09 09:33:36.769865 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:36.769560 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"dns-node-resolver\" with ErrImagePull: \"pull QPS exceeded\"" pod="openshift-dns/node-resolver-nx4l4" podUID="32307b23-721f-4ed0-add4-542ed80a29d8" Jul 09 09:33:37.007322 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:37.007242 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0652b01-6ddc-4820-8fbe-c2e5b643f663-metrics-certs\") pod \"network-metrics-daemon-wsdxs\" (UID: \"c0652b01-6ddc-4820-8fbe-c2e5b643f663\") " pod="openshift-multus/network-metrics-daemon-wsdxs" Jul 09 09:33:37.007322 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:37.007304 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a0c24dc3-5e21-4537-966b-0c5f9e46f8d9-original-pull-secret\") pod \"global-pull-secret-syncer-sw9zq\" (UID: \"a0c24dc3-5e21-4537-966b-0c5f9e46f8d9\") " pod="kube-system/global-pull-secret-syncer-sw9zq" Jul 09 09:33:37.007526 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:37.007417 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 09:33:37.007526 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:37.007442 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 09:33:37.007526 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:37.007489 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0652b01-6ddc-4820-8fbe-c2e5b643f663-metrics-certs podName:c0652b01-6ddc-4820-8fbe-c2e5b643f663 nodeName:}" failed. No retries permitted until 2026-07-09 09:33:38.007467787 +0000 UTC m=+3.099221686 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0652b01-6ddc-4820-8fbe-c2e5b643f663-metrics-certs") pod "network-metrics-daemon-wsdxs" (UID: "c0652b01-6ddc-4820-8fbe-c2e5b643f663") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 09:33:37.007526 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:37.007511 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0c24dc3-5e21-4537-966b-0c5f9e46f8d9-original-pull-secret podName:a0c24dc3-5e21-4537-966b-0c5f9e46f8d9 nodeName:}" failed. No retries permitted until 2026-07-09 09:33:38.007500951 +0000 UTC m=+3.099254854 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a0c24dc3-5e21-4537-966b-0c5f9e46f8d9-original-pull-secret") pod "global-pull-secret-syncer-sw9zq" (UID: "a0c24dc3-5e21-4537-966b-0c5f9e46f8d9") : object "kube-system"/"original-pull-secret" not registered Jul 09 09:33:37.108967 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:37.108343 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48bc9\" (UniqueName: \"kubernetes.io/projected/53e2e8a5-34d5-4317-9733-143a53880b9b-kube-api-access-48bc9\") pod \"network-check-target-gqlhc\" (UID: \"53e2e8a5-34d5-4317-9733-143a53880b9b\") " pod="openshift-network-diagnostics/network-check-target-gqlhc" Jul 09 09:33:37.108967 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:37.108509 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 09:33:37.108967 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:37.108531 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 09:33:37.108967 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:37.108543 2578 projected.go:196] Error preparing data for projected volume kube-api-access-48bc9 for pod openshift-network-diagnostics/network-check-target-gqlhc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 09:33:37.108967 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:37.108595 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/53e2e8a5-34d5-4317-9733-143a53880b9b-kube-api-access-48bc9 podName:53e2e8a5-34d5-4317-9733-143a53880b9b nodeName:}" failed. No retries permitted until 2026-07-09 09:33:38.10857802 +0000 UTC m=+3.200331952 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-48bc9" (UniqueName: "kubernetes.io/projected/53e2e8a5-34d5-4317-9733-143a53880b9b-kube-api-access-48bc9") pod "network-check-target-gqlhc" (UID: "53e2e8a5-34d5-4317-9733-143a53880b9b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 09:33:37.350390 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:37.350283 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 09:28:36 +0000 UTC" deadline="2028-02-19 05:16:51.796882341 +0000 UTC" Jul 09 09:33:37.350390 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:37.350343 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14155h43m14.446543405s" Jul 09 09:33:37.438989 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:37.438499 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wsdxs" Jul 09 09:33:37.438989 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:37.438626 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wsdxs" podUID="c0652b01-6ddc-4820-8fbe-c2e5b643f663" Jul 09 09:33:37.438989 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:37.438720 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gqlhc" Jul 09 09:33:37.438989 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:37.438813 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gqlhc" podUID="53e2e8a5-34d5-4317-9733-143a53880b9b" Jul 09 09:33:37.453083 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:37.452407 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nx4l4" event={"ID":"32307b23-721f-4ed0-add4-542ed80a29d8","Type":"ContainerStarted","Data":"14ff7be5d35e573349118ccd1f988bb5d58354bde7fb1be4fb5f56cacd19d746"} Jul 09 09:33:37.457521 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:37.457337 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"dns-node-resolver\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:28389dd118629aa9d7c2525202f3ad6359afb98b6a3bf7672cb1ecfae2418e23\\\": ErrImagePull: pull QPS exceeded\"" pod="openshift-dns/node-resolver-nx4l4" podUID="32307b23-721f-4ed0-add4-542ed80a29d8" Jul 09 09:33:37.474675 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:37.474647 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rspgp" event={"ID":"65d9652f-67de-405c-8e07-aa737e9205d8","Type":"ContainerStarted","Data":"1c4fda396b263eb3748af6f2af99dc561fd01d701ca37e48dac906fcdf2657c4"} Jul 09 09:33:37.482046 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:37.482019 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qbtkh" event={"ID":"f8587efa-7f0b-4a11-b03f-e08b942aa1f1","Type":"ContainerStarted","Data":"95af8d0e71dc951ff96563c4b41ccff704f5bbb18f69b5ae668bcb25f201f263"} Jul 09 09:33:37.489280 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:37.489208 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gbmbr" event={"ID":"dc248a4e-4aa6-4ea4-81b7-439e669b56c1","Type":"ContainerStarted","Data":"04e3c9c361afc44356bbf961b5874bfce4558abb66268da97785c77a3dd743f0"} Jul 09 09:33:37.498580 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:37.498531 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" event={"ID":"1677420d-7376-45ad-ac67-20128aec5756","Type":"ContainerStarted","Data":"a97a533a92431cae133127b67f3bf585505a444d9556577013a55e569fe7ba3c"} Jul 09 09:33:37.509806 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:37.509780 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-r6npq" event={"ID":"c7314861-eada-458d-bf61-21da88f8b1c8","Type":"ContainerStarted","Data":"818669a0bae20786785de96265cf1630c7dc6a1f8fb46ade3cafd96587f647d5"} Jul 09 09:33:37.515494 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:37.515467 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-clbg8" event={"ID":"6e7da002-bfa6-4527-bfb9-ee3bc36677f1","Type":"ContainerStarted","Data":"f418557b7904f348eba6a72b07619a6d204681be01c1e3960ac8cd575cfec868"} Jul 09 09:33:37.537826 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:37.537797 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z76mc" event={"ID":"f2d0ed74-687b-403d-9ad5-17ee9fc39d6a","Type":"ContainerStarted","Data":"72e52dadbdb74f343eeb3128e3943b6d16c4619ad4e735f7e1df7c71428e8e7e"} Jul 09 09:33:37.552341 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:37.552281 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dnzdl" event={"ID":"46a45a21-2ddc-4b0b-9988-911cde1598f7","Type":"ContainerStarted","Data":"4b8ee8a3df03802a8d0465facf2573cef977d911879757912af29cd570ce6af1"} Jul 09 09:33:37.561656 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:37.561628 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-204.ec2.internal" event={"ID":"f8f995f6fcf6abd8e78373eeaaaf3c86","Type":"ContainerStarted","Data":"0616572aa16433c7e591c2b0e019281b5f392607c0927a601c9fe1beb38dc3d7"} Jul 09 09:33:37.568993 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:37.568972 2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 09:33:37.580156 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:37.580128 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-204.ec2.internal" event={"ID":"4386692e678e561186646b99422b104a","Type":"ContainerStarted","Data":"b834a74edb8216d9b464c79b1607c338b4cf53e304f2a98d66551b80e1f4350c"} Jul 09 09:33:38.014767 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:38.014727 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a0c24dc3-5e21-4537-966b-0c5f9e46f8d9-original-pull-secret\") pod \"global-pull-secret-syncer-sw9zq\" (UID: \"a0c24dc3-5e21-4537-966b-0c5f9e46f8d9\") " pod="kube-system/global-pull-secret-syncer-sw9zq" Jul 09 09:33:38.014967 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:38.014832 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0652b01-6ddc-4820-8fbe-c2e5b643f663-metrics-certs\") pod \"network-metrics-daemon-wsdxs\" (UID: \"c0652b01-6ddc-4820-8fbe-c2e5b643f663\") " pod="openshift-multus/network-metrics-daemon-wsdxs" Jul 09 09:33:38.014967 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:38.014902 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 09:33:38.014967 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:38.014969 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0c24dc3-5e21-4537-966b-0c5f9e46f8d9-original-pull-secret podName:a0c24dc3-5e21-4537-966b-0c5f9e46f8d9 nodeName:}" failed. No retries permitted until 2026-07-09 09:33:40.01495003 +0000 UTC m=+5.106703939 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a0c24dc3-5e21-4537-966b-0c5f9e46f8d9-original-pull-secret") pod "global-pull-secret-syncer-sw9zq" (UID: "a0c24dc3-5e21-4537-966b-0c5f9e46f8d9") : object "kube-system"/"original-pull-secret" not registered Jul 09 09:33:38.015148 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:38.014999 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 09:33:38.015148 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:38.015051 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0652b01-6ddc-4820-8fbe-c2e5b643f663-metrics-certs podName:c0652b01-6ddc-4820-8fbe-c2e5b643f663 nodeName:}" failed. No retries permitted until 2026-07-09 09:33:40.015034577 +0000 UTC m=+5.106788496 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0652b01-6ddc-4820-8fbe-c2e5b643f663-metrics-certs") pod "network-metrics-daemon-wsdxs" (UID: "c0652b01-6ddc-4820-8fbe-c2e5b643f663") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 09:33:38.115991 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:38.115955 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48bc9\" (UniqueName: \"kubernetes.io/projected/53e2e8a5-34d5-4317-9733-143a53880b9b-kube-api-access-48bc9\") pod \"network-check-target-gqlhc\" (UID: \"53e2e8a5-34d5-4317-9733-143a53880b9b\") " pod="openshift-network-diagnostics/network-check-target-gqlhc" Jul 09 09:33:38.116166 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:38.116120 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 09:33:38.116166 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:38.116140 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 09:33:38.116166 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:38.116152 2578 projected.go:196] Error preparing data for projected volume kube-api-access-48bc9 for pod openshift-network-diagnostics/network-check-target-gqlhc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 09:33:38.116326 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:38.116205 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/53e2e8a5-34d5-4317-9733-143a53880b9b-kube-api-access-48bc9 podName:53e2e8a5-34d5-4317-9733-143a53880b9b nodeName:}" failed. No retries permitted until 2026-07-09 09:33:40.116188709 +0000 UTC m=+5.207942624 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-48bc9" (UniqueName: "kubernetes.io/projected/53e2e8a5-34d5-4317-9733-143a53880b9b-kube-api-access-48bc9") pod "network-check-target-gqlhc" (UID: "53e2e8a5-34d5-4317-9733-143a53880b9b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 09:33:38.351311 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:38.351017 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 09:28:36 +0000 UTC" deadline="2028-04-11 07:28:06.625070262 +0000 UTC" Jul 09 09:33:38.351311 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:38.351058 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15405h54m28.274018991s" Jul 09 09:33:38.437981 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:38.437950 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sw9zq" Jul 09 09:33:38.438147 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:38.438084 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sw9zq" podUID="a0c24dc3-5e21-4537-966b-0c5f9e46f8d9" Jul 09 09:33:39.222010 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:39.221736 2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 09:33:39.437908 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:39.437429 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wsdxs" Jul 09 09:33:39.437908 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:39.437565 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wsdxs" podUID="c0652b01-6ddc-4820-8fbe-c2e5b643f663" Jul 09 09:33:39.438934 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:39.438909 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gqlhc" Jul 09 09:33:39.439043 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:39.439023 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gqlhc" podUID="53e2e8a5-34d5-4317-9733-143a53880b9b" Jul 09 09:33:40.033193 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:40.033150 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a0c24dc3-5e21-4537-966b-0c5f9e46f8d9-original-pull-secret\") pod \"global-pull-secret-syncer-sw9zq\" (UID: \"a0c24dc3-5e21-4537-966b-0c5f9e46f8d9\") " pod="kube-system/global-pull-secret-syncer-sw9zq" Jul 09 09:33:40.033353 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:40.033257 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0652b01-6ddc-4820-8fbe-c2e5b643f663-metrics-certs\") pod \"network-metrics-daemon-wsdxs\" (UID: \"c0652b01-6ddc-4820-8fbe-c2e5b643f663\") " pod="openshift-multus/network-metrics-daemon-wsdxs" Jul 09 09:33:40.033397 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:40.033386 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 09:33:40.033480 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:40.033467 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0652b01-6ddc-4820-8fbe-c2e5b643f663-metrics-certs podName:c0652b01-6ddc-4820-8fbe-c2e5b643f663 nodeName:}" failed. No retries permitted until 2026-07-09 09:33:44.033429806 +0000 UTC m=+9.125183722 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0652b01-6ddc-4820-8fbe-c2e5b643f663-metrics-certs") pod "network-metrics-daemon-wsdxs" (UID: "c0652b01-6ddc-4820-8fbe-c2e5b643f663") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 09:33:40.033824 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:40.033810 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 09:33:40.033900 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:40.033868 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0c24dc3-5e21-4537-966b-0c5f9e46f8d9-original-pull-secret podName:a0c24dc3-5e21-4537-966b-0c5f9e46f8d9 nodeName:}" failed. No retries permitted until 2026-07-09 09:33:44.033858144 +0000 UTC m=+9.125612043 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a0c24dc3-5e21-4537-966b-0c5f9e46f8d9-original-pull-secret") pod "global-pull-secret-syncer-sw9zq" (UID: "a0c24dc3-5e21-4537-966b-0c5f9e46f8d9") : object "kube-system"/"original-pull-secret" not registered Jul 09 09:33:40.133870 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:40.133812 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48bc9\" (UniqueName: \"kubernetes.io/projected/53e2e8a5-34d5-4317-9733-143a53880b9b-kube-api-access-48bc9\") pod \"network-check-target-gqlhc\" (UID: \"53e2e8a5-34d5-4317-9733-143a53880b9b\") " pod="openshift-network-diagnostics/network-check-target-gqlhc" Jul 09 09:33:40.134060 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:40.134044 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 09:33:40.134120 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:40.134068 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 09:33:40.134120 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:40.134080 2578 projected.go:196] Error preparing data for projected volume kube-api-access-48bc9 for pod openshift-network-diagnostics/network-check-target-gqlhc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 09:33:40.134198 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:40.134132 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/53e2e8a5-34d5-4317-9733-143a53880b9b-kube-api-access-48bc9 podName:53e2e8a5-34d5-4317-9733-143a53880b9b nodeName:}" failed. No retries permitted until 2026-07-09 09:33:44.134117745 +0000 UTC m=+9.225871643 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-48bc9" (UniqueName: "kubernetes.io/projected/53e2e8a5-34d5-4317-9733-143a53880b9b-kube-api-access-48bc9") pod "network-check-target-gqlhc" (UID: "53e2e8a5-34d5-4317-9733-143a53880b9b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 09:33:40.437934 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:40.437853 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sw9zq" Jul 09 09:33:40.438369 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:40.437984 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sw9zq" podUID="a0c24dc3-5e21-4537-966b-0c5f9e46f8d9" Jul 09 09:33:41.440127 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:41.439409 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wsdxs" Jul 09 09:33:41.440127 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:41.439545 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wsdxs" podUID="c0652b01-6ddc-4820-8fbe-c2e5b643f663" Jul 09 09:33:41.440127 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:41.439975 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gqlhc" Jul 09 09:33:41.440127 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:41.440074 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gqlhc" podUID="53e2e8a5-34d5-4317-9733-143a53880b9b" Jul 09 09:33:42.438337 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:42.437610 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sw9zq" Jul 09 09:33:42.438337 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:42.437741 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sw9zq" podUID="a0c24dc3-5e21-4537-966b-0c5f9e46f8d9" Jul 09 09:33:43.438986 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:43.438948 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gqlhc" Jul 09 09:33:43.439402 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:43.439082 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gqlhc" podUID="53e2e8a5-34d5-4317-9733-143a53880b9b" Jul 09 09:33:43.439402 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:43.439156 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wsdxs" Jul 09 09:33:43.439402 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:43.439274 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wsdxs" podUID="c0652b01-6ddc-4820-8fbe-c2e5b643f663" Jul 09 09:33:44.064586 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:44.064549 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a0c24dc3-5e21-4537-966b-0c5f9e46f8d9-original-pull-secret\") pod \"global-pull-secret-syncer-sw9zq\" (UID: \"a0c24dc3-5e21-4537-966b-0c5f9e46f8d9\") " pod="kube-system/global-pull-secret-syncer-sw9zq" Jul 09 09:33:44.064780 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:44.064647 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0652b01-6ddc-4820-8fbe-c2e5b643f663-metrics-certs\") pod \"network-metrics-daemon-wsdxs\" (UID: \"c0652b01-6ddc-4820-8fbe-c2e5b643f663\") " pod="openshift-multus/network-metrics-daemon-wsdxs" Jul 09 09:33:44.064780 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:44.064770 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 09:33:44.064959 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:44.064831 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0652b01-6ddc-4820-8fbe-c2e5b643f663-metrics-certs podName:c0652b01-6ddc-4820-8fbe-c2e5b643f663 nodeName:}" failed. No retries permitted until 2026-07-09 09:33:52.06481257 +0000 UTC m=+17.156566470 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0652b01-6ddc-4820-8fbe-c2e5b643f663-metrics-certs") pod "network-metrics-daemon-wsdxs" (UID: "c0652b01-6ddc-4820-8fbe-c2e5b643f663") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 09:33:44.064959 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:44.064917 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 09:33:44.064959 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:44.064954 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0c24dc3-5e21-4537-966b-0c5f9e46f8d9-original-pull-secret podName:a0c24dc3-5e21-4537-966b-0c5f9e46f8d9 nodeName:}" failed. No retries permitted until 2026-07-09 09:33:52.0649421 +0000 UTC m=+17.156696003 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a0c24dc3-5e21-4537-966b-0c5f9e46f8d9-original-pull-secret") pod "global-pull-secret-syncer-sw9zq" (UID: "a0c24dc3-5e21-4537-966b-0c5f9e46f8d9") : object "kube-system"/"original-pull-secret" not registered Jul 09 09:33:44.166537 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:44.165920 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48bc9\" (UniqueName: \"kubernetes.io/projected/53e2e8a5-34d5-4317-9733-143a53880b9b-kube-api-access-48bc9\") pod \"network-check-target-gqlhc\" (UID: \"53e2e8a5-34d5-4317-9733-143a53880b9b\") " pod="openshift-network-diagnostics/network-check-target-gqlhc" Jul 09 09:33:44.166537 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:44.166105 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 09:33:44.166537 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:44.166127 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 09:33:44.166537 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:44.166140 2578 projected.go:196] Error preparing data for projected volume kube-api-access-48bc9 for pod openshift-network-diagnostics/network-check-target-gqlhc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 09:33:44.166537 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:44.166199 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/53e2e8a5-34d5-4317-9733-143a53880b9b-kube-api-access-48bc9 podName:53e2e8a5-34d5-4317-9733-143a53880b9b nodeName:}" failed. No retries permitted until 2026-07-09 09:33:52.166180478 +0000 UTC m=+17.257934393 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-48bc9" (UniqueName: "kubernetes.io/projected/53e2e8a5-34d5-4317-9733-143a53880b9b-kube-api-access-48bc9") pod "network-check-target-gqlhc" (UID: "53e2e8a5-34d5-4317-9733-143a53880b9b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 09:33:44.437251 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:44.437066 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sw9zq" Jul 09 09:33:44.437251 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:44.437191 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sw9zq" podUID="a0c24dc3-5e21-4537-966b-0c5f9e46f8d9" Jul 09 09:33:45.439666 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:45.438721 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wsdxs" Jul 09 09:33:45.439666 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:45.439098 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wsdxs" podUID="c0652b01-6ddc-4820-8fbe-c2e5b643f663" Jul 09 09:33:45.439666 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:45.439518 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gqlhc" Jul 09 09:33:45.439666 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:45.439622 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gqlhc" podUID="53e2e8a5-34d5-4317-9733-143a53880b9b" Jul 09 09:33:46.438075 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:46.437591 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sw9zq" Jul 09 09:33:46.438075 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:46.437723 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sw9zq" podUID="a0c24dc3-5e21-4537-966b-0c5f9e46f8d9" Jul 09 09:33:47.437283 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:47.437249 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wsdxs" Jul 09 09:33:47.437719 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:47.437253 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gqlhc" Jul 09 09:33:47.437719 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:47.437380 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wsdxs" podUID="c0652b01-6ddc-4820-8fbe-c2e5b643f663" Jul 09 09:33:47.437719 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:47.437478 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gqlhc" podUID="53e2e8a5-34d5-4317-9733-143a53880b9b" Jul 09 09:33:48.437233 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:48.437204 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sw9zq" Jul 09 09:33:48.437381 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:48.437308 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sw9zq" podUID="a0c24dc3-5e21-4537-966b-0c5f9e46f8d9" Jul 09 09:33:49.437219 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:49.437184 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wsdxs" Jul 09 09:33:49.437400 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:49.437307 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wsdxs" podUID="c0652b01-6ddc-4820-8fbe-c2e5b643f663" Jul 09 09:33:49.437400 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:49.437353 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gqlhc" Jul 09 09:33:49.437797 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:49.437463 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gqlhc" podUID="53e2e8a5-34d5-4317-9733-143a53880b9b" Jul 09 09:33:50.437599 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:50.437570 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sw9zq" Jul 09 09:33:50.438085 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:50.437675 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sw9zq" podUID="a0c24dc3-5e21-4537-966b-0c5f9e46f8d9" Jul 09 09:33:51.437904 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:51.437866 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gqlhc" Jul 09 09:33:51.437904 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:51.437899 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wsdxs" Jul 09 09:33:51.438341 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:51.437998 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gqlhc" podUID="53e2e8a5-34d5-4317-9733-143a53880b9b" Jul 09 09:33:51.438341 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:51.438132 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wsdxs" podUID="c0652b01-6ddc-4820-8fbe-c2e5b643f663" Jul 09 09:33:52.129193 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:52.129150 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a0c24dc3-5e21-4537-966b-0c5f9e46f8d9-original-pull-secret\") pod \"global-pull-secret-syncer-sw9zq\" (UID: \"a0c24dc3-5e21-4537-966b-0c5f9e46f8d9\") " pod="kube-system/global-pull-secret-syncer-sw9zq" Jul 09 09:33:52.129467 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:52.129232 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0652b01-6ddc-4820-8fbe-c2e5b643f663-metrics-certs\") pod \"network-metrics-daemon-wsdxs\" (UID: \"c0652b01-6ddc-4820-8fbe-c2e5b643f663\") " pod="openshift-multus/network-metrics-daemon-wsdxs" Jul 09 09:33:52.129467 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:52.129304 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 09:33:52.129467 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:52.129338 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 09:33:52.129467 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:52.129381 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0c24dc3-5e21-4537-966b-0c5f9e46f8d9-original-pull-secret podName:a0c24dc3-5e21-4537-966b-0c5f9e46f8d9 nodeName:}" failed. No retries permitted until 2026-07-09 09:34:08.129362783 +0000 UTC m=+33.221116699 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a0c24dc3-5e21-4537-966b-0c5f9e46f8d9-original-pull-secret") pod "global-pull-secret-syncer-sw9zq" (UID: "a0c24dc3-5e21-4537-966b-0c5f9e46f8d9") : object "kube-system"/"original-pull-secret" not registered Jul 09 09:33:52.129467 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:52.129403 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0652b01-6ddc-4820-8fbe-c2e5b643f663-metrics-certs podName:c0652b01-6ddc-4820-8fbe-c2e5b643f663 nodeName:}" failed. No retries permitted until 2026-07-09 09:34:08.129393127 +0000 UTC m=+33.221147026 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0652b01-6ddc-4820-8fbe-c2e5b643f663-metrics-certs") pod "network-metrics-daemon-wsdxs" (UID: "c0652b01-6ddc-4820-8fbe-c2e5b643f663") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 09:33:52.230209 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:52.230182 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48bc9\" (UniqueName: \"kubernetes.io/projected/53e2e8a5-34d5-4317-9733-143a53880b9b-kube-api-access-48bc9\") pod \"network-check-target-gqlhc\" (UID: \"53e2e8a5-34d5-4317-9733-143a53880b9b\") " pod="openshift-network-diagnostics/network-check-target-gqlhc" Jul 09 09:33:52.230368 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:52.230346 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 09:33:52.230422 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:52.230377 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 09:33:52.230422 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:52.230391 2578 projected.go:196] Error preparing data for projected volume kube-api-access-48bc9 for pod openshift-network-diagnostics/network-check-target-gqlhc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 09:33:52.230510 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:52.230449 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/53e2e8a5-34d5-4317-9733-143a53880b9b-kube-api-access-48bc9 podName:53e2e8a5-34d5-4317-9733-143a53880b9b nodeName:}" failed. No retries permitted until 2026-07-09 09:34:08.2304314 +0000 UTC m=+33.322185302 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-48bc9" (UniqueName: "kubernetes.io/projected/53e2e8a5-34d5-4317-9733-143a53880b9b-kube-api-access-48bc9") pod "network-check-target-gqlhc" (UID: "53e2e8a5-34d5-4317-9733-143a53880b9b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 09:33:52.438107 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:52.438023 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sw9zq" Jul 09 09:33:52.438496 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:52.438153 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sw9zq" podUID="a0c24dc3-5e21-4537-966b-0c5f9e46f8d9" Jul 09 09:33:53.437174 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:53.437140 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wsdxs" Jul 09 09:33:53.437345 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:53.437257 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wsdxs" podUID="c0652b01-6ddc-4820-8fbe-c2e5b643f663" Jul 09 09:33:53.437345 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:53.437317 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gqlhc" Jul 09 09:33:53.437459 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:53.437412 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gqlhc" podUID="53e2e8a5-34d5-4317-9733-143a53880b9b" Jul 09 09:33:54.437441 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:54.437411 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sw9zq" Jul 09 09:33:54.437918 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:54.437528 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sw9zq" podUID="a0c24dc3-5e21-4537-966b-0c5f9e46f8d9" Jul 09 09:33:55.437834 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:55.437804 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wsdxs" Jul 09 09:33:55.438305 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:55.437917 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wsdxs" podUID="c0652b01-6ddc-4820-8fbe-c2e5b643f663" Jul 09 09:33:55.438305 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:55.438011 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gqlhc" Jul 09 09:33:55.438305 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:55.438096 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gqlhc" podUID="53e2e8a5-34d5-4317-9733-143a53880b9b" Jul 09 09:33:56.437736 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:56.437707 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sw9zq" Jul 09 09:33:56.437900 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:56.437794 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sw9zq" podUID="a0c24dc3-5e21-4537-966b-0c5f9e46f8d9" Jul 09 09:33:57.441620 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:57.441446 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gqlhc" Jul 09 09:33:57.442236 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:57.441514 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wsdxs" Jul 09 09:33:57.442236 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:57.441690 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gqlhc" podUID="53e2e8a5-34d5-4317-9733-143a53880b9b" Jul 09 09:33:57.442236 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:57.441791 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wsdxs" podUID="c0652b01-6ddc-4820-8fbe-c2e5b643f663" Jul 09 09:33:57.617935 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:57.617902 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-clbg8" event={"ID":"6e7da002-bfa6-4527-bfb9-ee3bc36677f1","Type":"ContainerStarted","Data":"5cbfa4645a30ede7415c0ab8e24e55cf0511b9993c83d6bd342888252c082b8c"} Jul 09 09:33:57.619375 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:57.619354 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dnzdl" event={"ID":"46a45a21-2ddc-4b0b-9988-911cde1598f7","Type":"ContainerStarted","Data":"ba00861f9758cfe4595e667eedb313b67c18a9e2388cf6f67c9447be638bd1fc"} Jul 09 09:33:57.620672 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:57.620651 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-204.ec2.internal" event={"ID":"4386692e678e561186646b99422b104a","Type":"ContainerStarted","Data":"17b082ff90d6fda09cfc3bb71b06331e34e8999c2b896d6f670410c165895550"} Jul 09 09:33:57.624987 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:57.624970 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xf4sw_1677420d-7376-45ad-ac67-20128aec5756/ovn-acl-logging/0.log" Jul 09 09:33:57.625304 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:57.625287 2578 generic.go:356] "Generic (PLEG): container finished" podID="1677420d-7376-45ad-ac67-20128aec5756" containerID="1124e4d31de94686c1f1f0c51b82618749f9b6f49c1e332ad1e5f7f79072f6e4" exitCode=1 Jul 09 09:33:57.625365 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:57.625316 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" event={"ID":"1677420d-7376-45ad-ac67-20128aec5756","Type":"ContainerStarted","Data":"fc0334ad0c2b21ab699fc096542837393cbfc41bd1ded07260f0a252054b9ae8"} Jul 09 09:33:57.625365 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:57.625358 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" event={"ID":"1677420d-7376-45ad-ac67-20128aec5756","Type":"ContainerStarted","Data":"b29a31df9deb55a852641b65acb3383916f11befc7c8b6b2143721971f02dd9d"} Jul 09 09:33:57.625438 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:57.625386 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" event={"ID":"1677420d-7376-45ad-ac67-20128aec5756","Type":"ContainerStarted","Data":"a3b8d24feeeadd4eaefc12a6f7b0d4736274a32c08f2b16917a31b751484fcf0"} Jul 09 09:33:57.625438 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:57.625413 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" event={"ID":"1677420d-7376-45ad-ac67-20128aec5756","Type":"ContainerStarted","Data":"82576f1619839696e816a171d3c43923e01f81722c3f7d297cff2931f441e2ad"} Jul 09 09:33:57.625503 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:57.625439 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" event={"ID":"1677420d-7376-45ad-ac67-20128aec5756","Type":"ContainerDied","Data":"1124e4d31de94686c1f1f0c51b82618749f9b6f49c1e332ad1e5f7f79072f6e4"} Jul 09 09:33:57.625503 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:57.625469 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" event={"ID":"1677420d-7376-45ad-ac67-20128aec5756","Type":"ContainerStarted","Data":"0353cc43493c7dd7be1600cfa4a4a482a5469fe8cf31e30b9c6ba1d7ef43f333"} Jul 09 09:33:57.668736 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:57.668683 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-clbg8" podStartSLOduration=2.193873063 podStartE2EDuration="22.668666266s" totalImagesPullingTime="20.474793203s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 09:33:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:33:57.642034653 +0000 UTC m=+22.733788575" watchObservedRunningTime="2026-07-09 09:33:57.668666266 +0000 UTC m=+22.760420188" Jul 09 09:33:57.668981 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:57.668952 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-dnzdl" podStartSLOduration=2.540190963 podStartE2EDuration="22.668940412s" totalImagesPullingTime="20.128749449s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 09:33:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:33:57.668347689 +0000 UTC m=+22.760101611" watchObservedRunningTime="2026-07-09 09:33:57.668940412 +0000 UTC m=+22.760694333" Jul 09 09:33:58.437296 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:58.437073 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sw9zq" Jul 09 09:33:58.437488 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:58.437356 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sw9zq" podUID="a0c24dc3-5e21-4537-966b-0c5f9e46f8d9" Jul 09 09:33:58.631494 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:58.631457 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nx4l4" event={"ID":"32307b23-721f-4ed0-add4-542ed80a29d8","Type":"ContainerStarted","Data":"c540d4602c9f221d7bb9e5e41b9ab4529b78c1d6f57ac4801e9bc53244844823"} Jul 09 09:33:58.632928 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:58.632902 2578 generic.go:356] "Generic (PLEG): container finished" podID="65d9652f-67de-405c-8e07-aa737e9205d8" containerID="b737e899a8715860d32aa712c610a66814812096ade4fbbe58e97b1088eb59d5" exitCode=0 Jul 09 09:33:58.633031 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:58.632965 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rspgp" event={"ID":"65d9652f-67de-405c-8e07-aa737e9205d8","Type":"ContainerDied","Data":"b737e899a8715860d32aa712c610a66814812096ade4fbbe58e97b1088eb59d5"} Jul 09 09:33:58.634262 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:58.634235 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qbtkh" event={"ID":"f8587efa-7f0b-4a11-b03f-e08b942aa1f1","Type":"ContainerStarted","Data":"97b87da862d6422744036107beb0e90a9f9bd4c9cfde363a91d0c3a685e5aff5"} Jul 09 09:33:58.635672 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:58.635652 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gbmbr" event={"ID":"dc248a4e-4aa6-4ea4-81b7-439e669b56c1","Type":"ContainerStarted","Data":"cc5e44d7609f72c49c059fa7054692930a37f9b22b8acf7951814bfea7a4e82f"} Jul 09 09:33:58.637039 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:58.637021 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-r6npq" event={"ID":"c7314861-eada-458d-bf61-21da88f8b1c8","Type":"ContainerStarted","Data":"8793bc0c9460c650112b488e0d2e42266f90dcb93e29502afadbd3170430e968"} Jul 09 09:33:58.638306 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:58.638287 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z76mc" event={"ID":"f2d0ed74-687b-403d-9ad5-17ee9fc39d6a","Type":"ContainerStarted","Data":"bdb0854b5e56e2ed64c385de50df267b02180b4ace60a26a8025dadd9a3c4ead"} Jul 09 09:33:58.639707 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:58.639689 2578 generic.go:356] "Generic (PLEG): container finished" podID="f8f995f6fcf6abd8e78373eeaaaf3c86" containerID="e5ab66e5de96a85075ff6b193a045b5f2034360636ac91487d797d3b40c97156" exitCode=0 Jul 09 09:33:58.639802 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:58.639745 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-204.ec2.internal" event={"ID":"f8f995f6fcf6abd8e78373eeaaaf3c86","Type":"ContainerDied","Data":"e5ab66e5de96a85075ff6b193a045b5f2034360636ac91487d797d3b40c97156"} Jul 09 09:33:58.653990 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:58.653952 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-204.ec2.internal" podStartSLOduration=22.653940813 podStartE2EDuration="22.653940813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 09:33:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:33:57.686074963 +0000 UTC m=+22.777828883" watchObservedRunningTime="2026-07-09 09:33:58.653940813 +0000 UTC m=+23.745694785" Jul 09 09:33:58.671491 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:58.671458 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-nx4l4" podStartSLOduration=23.671446225 podStartE2EDuration="23.671446225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 09:33:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=1 observedRunningTime="2026-07-09 09:33:58.654174033 +0000 UTC m=+23.745927952" watchObservedRunningTime="2026-07-09 09:33:58.671446225 +0000 UTC m=+23.763200203" Jul 09 09:33:58.713286 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:58.713245 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-r6npq" podStartSLOduration=3.715162647 podStartE2EDuration="23.713228177s" totalImagesPullingTime="19.99806553s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 09:33:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:33:58.691971097 +0000 UTC m=+23.783725041" watchObservedRunningTime="2026-07-09 09:33:58.713228177 +0000 UTC m=+23.804982095" Jul 09 09:33:58.746754 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:58.746722 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-gbmbr" podStartSLOduration=3.783775274 podStartE2EDuration="23.746703156s" totalImagesPullingTime="19.962927882s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:33:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:33:58.714019734 +0000 UTC m=+23.805773654" watchObservedRunningTime="2026-07-09 09:33:58.746703156 +0000 UTC m=+23.838457075" Jul 09 09:33:58.766380 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:58.766342 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-qbtkh" podStartSLOduration=3.841580585 podStartE2EDuration="23.766328058s" totalImagesPullingTime="19.924747473s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 09:33:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:33:58.764994005 +0000 UTC m=+23.856747950" watchObservedRunningTime="2026-07-09 09:33:58.766328058 +0000 UTC m=+23.858081978" Jul 09 09:33:58.891621 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:58.891588 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gbmbr_dc248a4e-4aa6-4ea4-81b7-439e669b56c1/konnectivity-agent/0.log" limit=1024 Jul 09 09:33:59.079431 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:59.079411 2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 09 09:33:59.364373 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:59.364242 2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-09T09:33:59.079427628Z","UUID":"715f7cdf-b627-4d42-8ce7-ab043eebdbb8","Handler":null,"Name":"","Endpoint":""} Jul 09 09:33:59.365646 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:59.365627 2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 09 09:33:59.365756 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:59.365669 2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 09 09:33:59.440749 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:59.440724 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gqlhc" Jul 09 09:33:59.440930 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:59.440831 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gqlhc" podUID="53e2e8a5-34d5-4317-9733-143a53880b9b" Jul 09 09:33:59.440930 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:59.440878 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wsdxs" Jul 09 09:33:59.441032 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:33:59.440946 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wsdxs" podUID="c0652b01-6ddc-4820-8fbe-c2e5b643f663" Jul 09 09:33:59.644415 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:59.644246 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-204.ec2.internal" event={"ID":"f8f995f6fcf6abd8e78373eeaaaf3c86","Type":"ContainerStarted","Data":"8df4aebac0a85c1e5703922e2721f56e60f6bf88de06766c699681e9f1a53ab0"} Jul 09 09:33:59.647131 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:59.647105 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z76mc" event={"ID":"f2d0ed74-687b-403d-9ad5-17ee9fc39d6a","Type":"ContainerStarted","Data":"2f06a1dfd8beecbf261d37824e3c86f80e72cad3578bfe6b284ca51e18113cf8"} Jul 09 09:33:59.660626 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:33:59.660577 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-204.ec2.internal" podStartSLOduration=23.660559641 podStartE2EDuration="23.660559641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 09:33:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:33:59.660347493 +0000 UTC m=+24.752101414" watchObservedRunningTime="2026-07-09 09:33:59.660559641 +0000 UTC m=+24.752313561" Jul 09 09:34:00.437279 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:00.437247 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sw9zq" Jul 09 09:34:00.437429 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:34:00.437378 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sw9zq" podUID="a0c24dc3-5e21-4537-966b-0c5f9e46f8d9" Jul 09 09:34:00.652886 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:00.652831 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xf4sw_1677420d-7376-45ad-ac67-20128aec5756/ovn-acl-logging/0.log" Jul 09 09:34:00.653326 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:00.653293 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" event={"ID":"1677420d-7376-45ad-ac67-20128aec5756","Type":"ContainerStarted","Data":"66c8ce7656d2fd09bafca232ec9e18b1cb4d422716b32be48a6fac25d5797f83"} Jul 09 09:34:00.655726 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:00.655705 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z76mc" event={"ID":"f2d0ed74-687b-403d-9ad5-17ee9fc39d6a","Type":"ContainerStarted","Data":"b62a4c2d224d61fe49ffb3c67f9f783a013736f3d9c981b7246162d257ffcf20"} Jul 09 09:34:00.677213 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:00.677170 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z76mc" podStartSLOduration=3.260101314 podStartE2EDuration="25.677158715s" totalImagesPullingTime="22.417057401s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 09:33:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:34:00.677026027 +0000 UTC m=+25.768779949" watchObservedRunningTime="2026-07-09 09:34:00.677158715 +0000 UTC m=+25.768912634" Jul 09 09:34:01.437957 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:01.437927 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wsdxs" Jul 09 09:34:01.438130 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:34:01.438033 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wsdxs" podUID="c0652b01-6ddc-4820-8fbe-c2e5b643f663" Jul 09 09:34:01.438130 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:01.438085 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gqlhc" Jul 09 09:34:01.438254 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:34:01.438170 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gqlhc" podUID="53e2e8a5-34d5-4317-9733-143a53880b9b" Jul 09 09:34:02.438093 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:02.437876 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sw9zq" Jul 09 09:34:02.438696 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:34:02.438196 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sw9zq" podUID="a0c24dc3-5e21-4537-966b-0c5f9e46f8d9" Jul 09 09:34:02.663454 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:02.663427 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xf4sw_1677420d-7376-45ad-ac67-20128aec5756/ovn-acl-logging/0.log" Jul 09 09:34:02.663904 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:02.663866 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" event={"ID":"1677420d-7376-45ad-ac67-20128aec5756","Type":"ContainerStarted","Data":"5a232c3f7f4b8ddf16acc6646e0ba0b98fa519b21e7cc07a4f6d5727b1ab5215"} Jul 09 09:34:02.664389 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:02.664365 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" Jul 09 09:34:02.664477 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:02.664442 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" Jul 09 09:34:02.664477 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:02.664376 2578 scope.go:117] "RemoveContainer" containerID="1124e4d31de94686c1f1f0c51b82618749f9b6f49c1e332ad1e5f7f79072f6e4" Jul 09 09:34:02.681693 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:02.681671 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" Jul 09 09:34:02.682780 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:02.682762 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" Jul 09 09:34:03.308621 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:03.308569 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-gbmbr" Jul 09 09:34:03.309381 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:03.309354 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-gbmbr" Jul 09 09:34:03.437589 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:03.437517 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wsdxs" Jul 09 09:34:03.437589 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:03.437529 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gqlhc" Jul 09 09:34:03.437783 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:34:03.437641 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wsdxs" podUID="c0652b01-6ddc-4820-8fbe-c2e5b643f663" Jul 09 09:34:03.437820 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:34:03.437777 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gqlhc" podUID="53e2e8a5-34d5-4317-9733-143a53880b9b" Jul 09 09:34:03.665596 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:03.665576 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-gbmbr" Jul 09 09:34:03.666229 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:03.666213 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-gbmbr" Jul 09 09:34:04.437811 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:04.437790 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sw9zq" Jul 09 09:34:04.437962 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:34:04.437900 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sw9zq" podUID="a0c24dc3-5e21-4537-966b-0c5f9e46f8d9" Jul 09 09:34:04.669658 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:04.669629 2578 generic.go:356] "Generic (PLEG): container finished" podID="65d9652f-67de-405c-8e07-aa737e9205d8" containerID="5da17fc259efa56c69e1447c2048c8172df5e2d7ab8756d07486c40cb36b59a4" exitCode=0 Jul 09 09:34:04.670059 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:04.669676 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rspgp" event={"ID":"65d9652f-67de-405c-8e07-aa737e9205d8","Type":"ContainerDied","Data":"5da17fc259efa56c69e1447c2048c8172df5e2d7ab8756d07486c40cb36b59a4"} Jul 09 09:34:04.673485 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:04.673468 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xf4sw_1677420d-7376-45ad-ac67-20128aec5756/ovn-acl-logging/0.log" Jul 09 09:34:04.674010 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:04.673986 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" event={"ID":"1677420d-7376-45ad-ac67-20128aec5756","Type":"ContainerStarted","Data":"de3597e8492260a4ec96cee2edf33035914a0aaf8f7fcd04d43a517aac389fa2"} Jul 09 09:34:04.674010 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:04.674003 2578 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 09 09:34:04.732155 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:04.732085 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" podStartSLOduration=9.524413705 podStartE2EDuration="29.73207423s" totalImagesPullingTime="20.207660525s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 09:33:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:34:04.730953424 +0000 UTC m=+29.822707344" watchObservedRunningTime="2026-07-09 09:34:04.73207423 +0000 UTC m=+29.823828149" Jul 09 09:34:05.073961 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:05.073894 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" Jul 09 09:34:05.438556 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:05.438521 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wsdxs" Jul 09 09:34:05.438731 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:34:05.438673 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wsdxs" podUID="c0652b01-6ddc-4820-8fbe-c2e5b643f663" Jul 09 09:34:05.438790 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:05.438729 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gqlhc" Jul 09 09:34:05.438870 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:34:05.438817 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gqlhc" podUID="53e2e8a5-34d5-4317-9733-143a53880b9b" Jul 09 09:34:05.678503 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:05.678326 2578 generic.go:356] "Generic (PLEG): container finished" podID="65d9652f-67de-405c-8e07-aa737e9205d8" containerID="b4a11e94d8facbbf36ba213bd01bcc919b1c20e945c3c4a478f6d4fdc3fd35db" exitCode=0 Jul 09 09:34:05.678815 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:05.678403 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rspgp" event={"ID":"65d9652f-67de-405c-8e07-aa737e9205d8","Type":"ContainerDied","Data":"b4a11e94d8facbbf36ba213bd01bcc919b1c20e945c3c4a478f6d4fdc3fd35db"} Jul 09 09:34:06.437087 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:06.437057 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sw9zq" Jul 09 09:34:06.437238 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:34:06.437155 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sw9zq" podUID="a0c24dc3-5e21-4537-966b-0c5f9e46f8d9" Jul 09 09:34:06.682471 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:06.682442 2578 generic.go:356] "Generic (PLEG): container finished" podID="65d9652f-67de-405c-8e07-aa737e9205d8" containerID="78af19a630a870b2b057e3a25d4c25dea759dce0c913ae71da283e3c3cb3df8c" exitCode=0 Jul 09 09:34:06.682751 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:06.682528 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rspgp" event={"ID":"65d9652f-67de-405c-8e07-aa737e9205d8","Type":"ContainerDied","Data":"78af19a630a870b2b057e3a25d4c25dea759dce0c913ae71da283e3c3cb3df8c"} Jul 09 09:34:06.692181 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:06.692093 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" podUID="1677420d-7376-45ad-ac67-20128aec5756" containerName="ovnkube-controller" probeResult="failure" output="" Jul 09 09:34:07.438028 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:07.437995 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wsdxs" Jul 09 09:34:07.438210 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:34:07.438130 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wsdxs" podUID="c0652b01-6ddc-4820-8fbe-c2e5b643f663" Jul 09 09:34:07.438518 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:07.438491 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gqlhc" Jul 09 09:34:07.438654 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:34:07.438571 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gqlhc" podUID="53e2e8a5-34d5-4317-9733-143a53880b9b" Jul 09 09:34:08.146604 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:08.146568 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0652b01-6ddc-4820-8fbe-c2e5b643f663-metrics-certs\") pod \"network-metrics-daemon-wsdxs\" (UID: \"c0652b01-6ddc-4820-8fbe-c2e5b643f663\") " pod="openshift-multus/network-metrics-daemon-wsdxs" Jul 09 09:34:08.147031 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:08.146623 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a0c24dc3-5e21-4537-966b-0c5f9e46f8d9-original-pull-secret\") pod \"global-pull-secret-syncer-sw9zq\" (UID: \"a0c24dc3-5e21-4537-966b-0c5f9e46f8d9\") " pod="kube-system/global-pull-secret-syncer-sw9zq" Jul 09 09:34:08.147031 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:34:08.146727 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 09:34:08.147031 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:34:08.146751 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 09:34:08.147031 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:34:08.146804 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0c24dc3-5e21-4537-966b-0c5f9e46f8d9-original-pull-secret podName:a0c24dc3-5e21-4537-966b-0c5f9e46f8d9 nodeName:}" failed. No retries permitted until 2026-07-09 09:34:40.146784404 +0000 UTC m=+65.238538307 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a0c24dc3-5e21-4537-966b-0c5f9e46f8d9-original-pull-secret") pod "global-pull-secret-syncer-sw9zq" (UID: "a0c24dc3-5e21-4537-966b-0c5f9e46f8d9") : object "kube-system"/"original-pull-secret" not registered Jul 09 09:34:08.147031 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:34:08.146821 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0652b01-6ddc-4820-8fbe-c2e5b643f663-metrics-certs podName:c0652b01-6ddc-4820-8fbe-c2e5b643f663 nodeName:}" failed. No retries permitted until 2026-07-09 09:34:40.146811981 +0000 UTC m=+65.238565884 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0652b01-6ddc-4820-8fbe-c2e5b643f663-metrics-certs") pod "network-metrics-daemon-wsdxs" (UID: "c0652b01-6ddc-4820-8fbe-c2e5b643f663") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 09:34:08.247668 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:08.247641 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48bc9\" (UniqueName: \"kubernetes.io/projected/53e2e8a5-34d5-4317-9733-143a53880b9b-kube-api-access-48bc9\") pod \"network-check-target-gqlhc\" (UID: \"53e2e8a5-34d5-4317-9733-143a53880b9b\") " pod="openshift-network-diagnostics/network-check-target-gqlhc" Jul 09 09:34:08.247822 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:34:08.247771 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 09:34:08.247822 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:34:08.247794 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 09:34:08.247822 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:34:08.247807 2578 projected.go:196] Error preparing data for projected volume kube-api-access-48bc9 for pod openshift-network-diagnostics/network-check-target-gqlhc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 09:34:08.247989 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:34:08.247882 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/53e2e8a5-34d5-4317-9733-143a53880b9b-kube-api-access-48bc9 podName:53e2e8a5-34d5-4317-9733-143a53880b9b nodeName:}" failed. No retries permitted until 2026-07-09 09:34:40.24786416 +0000 UTC m=+65.339618074 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-48bc9" (UniqueName: "kubernetes.io/projected/53e2e8a5-34d5-4317-9733-143a53880b9b-kube-api-access-48bc9") pod "network-check-target-gqlhc" (UID: "53e2e8a5-34d5-4317-9733-143a53880b9b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 09:34:08.437936 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:08.437868 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sw9zq" Jul 09 09:34:08.438080 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:34:08.437980 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sw9zq" podUID="a0c24dc3-5e21-4537-966b-0c5f9e46f8d9" Jul 09 09:34:09.437547 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:09.437512 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wsdxs" Jul 09 09:34:09.438047 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:34:09.437678 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wsdxs" podUID="c0652b01-6ddc-4820-8fbe-c2e5b643f663" Jul 09 09:34:09.438047 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:09.437741 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gqlhc" Jul 09 09:34:09.438047 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:34:09.437861 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gqlhc" podUID="53e2e8a5-34d5-4317-9733-143a53880b9b" Jul 09 09:34:10.437660 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:10.437620 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sw9zq" Jul 09 09:34:10.438072 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:34:10.437761 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sw9zq" podUID="a0c24dc3-5e21-4537-966b-0c5f9e46f8d9" Jul 09 09:34:11.437609 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:11.437581 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wsdxs" Jul 09 09:34:11.437856 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:34:11.437703 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wsdxs" podUID="c0652b01-6ddc-4820-8fbe-c2e5b643f663" Jul 09 09:34:11.437856 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:11.437756 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gqlhc" Jul 09 09:34:11.438226 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:34:11.437856 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gqlhc" podUID="53e2e8a5-34d5-4317-9733-143a53880b9b" Jul 09 09:34:11.695707 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:11.695630 2578 generic.go:356] "Generic (PLEG): container finished" podID="65d9652f-67de-405c-8e07-aa737e9205d8" containerID="45bb8ed827a0ed5709a19709fec38cf13c82761ac9d0f1e91df84c94cd4d7d2d" exitCode=0 Jul 09 09:34:11.695707 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:11.695688 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rspgp" event={"ID":"65d9652f-67de-405c-8e07-aa737e9205d8","Type":"ContainerDied","Data":"45bb8ed827a0ed5709a19709fec38cf13c82761ac9d0f1e91df84c94cd4d7d2d"} Jul 09 09:34:12.437763 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:12.437732 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sw9zq" Jul 09 09:34:12.437974 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:34:12.437871 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sw9zq" podUID="a0c24dc3-5e21-4537-966b-0c5f9e46f8d9" Jul 09 09:34:12.700297 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:12.700228 2578 generic.go:356] "Generic (PLEG): container finished" podID="65d9652f-67de-405c-8e07-aa737e9205d8" containerID="853444c993d44cc7bec33ad470839892584d358a7537549f15abde03656b4e16" exitCode=0 Jul 09 09:34:12.700297 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:12.700266 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rspgp" event={"ID":"65d9652f-67de-405c-8e07-aa737e9205d8","Type":"ContainerDied","Data":"853444c993d44cc7bec33ad470839892584d358a7537549f15abde03656b4e16"} Jul 09 09:34:13.438084 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:13.438055 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wsdxs" Jul 09 09:34:13.438488 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:13.438108 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gqlhc" Jul 09 09:34:13.438488 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:34:13.438175 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gqlhc" podUID="53e2e8a5-34d5-4317-9733-143a53880b9b" Jul 09 09:34:13.438488 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:34:13.438250 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wsdxs" podUID="c0652b01-6ddc-4820-8fbe-c2e5b643f663" Jul 09 09:34:13.705889 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:13.705805 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rspgp" event={"ID":"65d9652f-67de-405c-8e07-aa737e9205d8","Type":"ContainerStarted","Data":"040257eed96d8299e4ed2c032f2de74569170722d66f4ca139d0c371f8788c95"} Jul 09 09:34:13.731653 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:13.731609 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-rspgp" podStartSLOduration=7.570427939 podStartE2EDuration="38.731596426s" totalImagesPullingTime="31.161168487s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 09:33:35 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:34:13.729984781 +0000 UTC m=+38.821738701" watchObservedRunningTime="2026-07-09 09:34:13.731596426 +0000 UTC m=+38.823350345" Jul 09 09:34:14.258792 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:14.258750 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sw9zq"] Jul 09 09:34:14.258982 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:14.258954 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sw9zq" Jul 09 09:34:14.259116 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:34:14.259087 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sw9zq" podUID="a0c24dc3-5e21-4537-966b-0c5f9e46f8d9" Jul 09 09:34:14.262990 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:14.262943 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wsdxs"] Jul 09 09:34:14.263159 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:14.263056 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wsdxs" Jul 09 09:34:14.263225 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:34:14.263169 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wsdxs" podUID="c0652b01-6ddc-4820-8fbe-c2e5b643f663" Jul 09 09:34:14.263792 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:14.263762 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gqlhc"] Jul 09 09:34:14.263912 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:14.263879 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gqlhc" Jul 09 09:34:14.263983 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:34:14.263965 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gqlhc" podUID="53e2e8a5-34d5-4317-9733-143a53880b9b" Jul 09 09:34:15.438857 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:15.438680 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gqlhc" Jul 09 09:34:15.439275 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:34:15.438915 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gqlhc" podUID="53e2e8a5-34d5-4317-9733-143a53880b9b" Jul 09 09:34:15.439275 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:15.438757 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wsdxs" Jul 09 09:34:15.439275 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:34:15.439066 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wsdxs" podUID="c0652b01-6ddc-4820-8fbe-c2e5b643f663" Jul 09 09:34:16.437283 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:16.437254 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sw9zq" Jul 09 09:34:16.437430 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:34:16.437346 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sw9zq" podUID="a0c24dc3-5e21-4537-966b-0c5f9e46f8d9" Jul 09 09:34:17.437297 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:17.437264 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wsdxs" Jul 09 09:34:17.437731 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:34:17.437386 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wsdxs" podUID="c0652b01-6ddc-4820-8fbe-c2e5b643f663" Jul 09 09:34:17.437731 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:17.437434 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gqlhc" Jul 09 09:34:17.437731 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:34:17.437509 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gqlhc" podUID="53e2e8a5-34d5-4317-9733-143a53880b9b" Jul 09 09:34:18.437218 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:18.437189 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sw9zq" Jul 09 09:34:18.437389 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:34:18.437281 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sw9zq" podUID="a0c24dc3-5e21-4537-966b-0c5f9e46f8d9" Jul 09 09:34:18.654224 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:18.654202 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-204.ec2.internal" event="NodeReady" Jul 09 09:34:18.654340 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:18.654299 2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 09 09:34:18.702347 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:18.702288 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-ztphn"] Jul 09 09:34:18.716578 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:18.716558 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ztphn" Jul 09 09:34:18.716578 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:18.716561 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-456st"] Jul 09 09:34:18.719156 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:18.719136 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 09 09:34:18.719255 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:18.719223 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-mlrpb\"" Jul 09 09:34:18.719315 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:18.719299 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 09 09:34:18.726647 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:18.726626 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ztphn"] Jul 09 09:34:18.726739 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:18.726665 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-456st"] Jul 09 09:34:18.726739 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:18.726704 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-cbnbv"] Jul 09 09:34:18.726739 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:18.726726 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-456st" Jul 09 09:34:18.729168 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:18.729149 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 09 09:34:18.729463 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:18.729450 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 09 09:34:18.729463 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:18.729455 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-sxdgx\"" Jul 09 09:34:18.729644 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:18.729630 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 09 09:34:18.729785 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:18.729769 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 09 09:34:18.752657 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:18.752636 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cbnbv"] Jul 09 09:34:18.752741 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:18.752713 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cbnbv" Jul 09 09:34:18.758871 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:18.758827 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 09 09:34:18.758967 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:18.758887 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-ckfzz\"" Jul 09 09:34:18.759041 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:18.759025 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 09 09:34:18.759111 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:18.759098 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 09 09:34:18.835617 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:18.835591 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f5bf5aed-640a-478f-a004-94b74b6d3ed9-metrics-tls\") pod \"dns-default-ztphn\" (UID: \"f5bf5aed-640a-478f-a004-94b74b6d3ed9\") " pod="openshift-dns/dns-default-ztphn" Jul 09 09:34:18.835743 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:18.835618 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dffd25e2-f81c-4c5f-afeb-1b56c2930e68-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-456st\" (UID: \"dffd25e2-f81c-4c5f-afeb-1b56c2930e68\") " pod="openshift-insights/insights-runtime-extractor-456st" Jul 09 09:34:18.835743 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:18.835641 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhx47\" (UniqueName: \"kubernetes.io/projected/dffd25e2-f81c-4c5f-afeb-1b56c2930e68-kube-api-access-bhx47\") pod \"insights-runtime-extractor-456st\" (UID: \"dffd25e2-f81c-4c5f-afeb-1b56c2930e68\") " pod="openshift-insights/insights-runtime-extractor-456st" Jul 09 09:34:18.835743 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:18.835672 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/dffd25e2-f81c-4c5f-afeb-1b56c2930e68-crio-socket\") pod \"insights-runtime-extractor-456st\" (UID: \"dffd25e2-f81c-4c5f-afeb-1b56c2930e68\") " pod="openshift-insights/insights-runtime-extractor-456st" Jul 09 09:34:18.835743 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:18.835692 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f5bf5aed-640a-478f-a004-94b74b6d3ed9-config-volume\") pod \"dns-default-ztphn\" (UID: \"f5bf5aed-640a-478f-a004-94b74b6d3ed9\") " pod="openshift-dns/dns-default-ztphn" Jul 09 09:34:18.835743 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:18.835731 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dffd25e2-f81c-4c5f-afeb-1b56c2930e68-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-456st\" (UID: \"dffd25e2-f81c-4c5f-afeb-1b56c2930e68\") " pod="openshift-insights/insights-runtime-extractor-456st" Jul 09 09:34:18.835987 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:18.835784 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/dffd25e2-f81c-4c5f-afeb-1b56c2930e68-data-volume\") pod \"insights-runtime-extractor-456st\" (UID: \"dffd25e2-f81c-4c5f-afeb-1b56c2930e68\") " pod="openshift-insights/insights-runtime-extractor-456st" Jul 09 09:34:18.835987 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:18.835853 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgl9z\" (UniqueName: \"kubernetes.io/projected/f5bf5aed-640a-478f-a004-94b74b6d3ed9-kube-api-access-fgl9z\") pod \"dns-default-ztphn\" (UID: \"f5bf5aed-640a-478f-a004-94b74b6d3ed9\") " pod="openshift-dns/dns-default-ztphn" Jul 09 09:34:18.835987 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:18.835888 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5bf5aed-640a-478f-a004-94b74b6d3ed9-tmp-dir\") pod \"dns-default-ztphn\" (UID: \"f5bf5aed-640a-478f-a004-94b74b6d3ed9\") " pod="openshift-dns/dns-default-ztphn" Jul 09 09:34:18.936951 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:18.936924 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgl9z\" (UniqueName: \"kubernetes.io/projected/f5bf5aed-640a-478f-a004-94b74b6d3ed9-kube-api-access-fgl9z\") pod \"dns-default-ztphn\" (UID: \"f5bf5aed-640a-478f-a004-94b74b6d3ed9\") " pod="openshift-dns/dns-default-ztphn" Jul 09 09:34:18.937095 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:18.936954 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5bf5aed-640a-478f-a004-94b74b6d3ed9-tmp-dir\") pod \"dns-default-ztphn\" (UID: \"f5bf5aed-640a-478f-a004-94b74b6d3ed9\") " pod="openshift-dns/dns-default-ztphn" Jul 09 09:34:18.937095 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:18.936983 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47946\" (UniqueName: \"kubernetes.io/projected/6a11f547-4bad-4cc3-9a49-bc04f9eac0e9-kube-api-access-47946\") pod \"ingress-canary-cbnbv\" (UID: \"6a11f547-4bad-4cc3-9a49-bc04f9eac0e9\") " pod="openshift-ingress-canary/ingress-canary-cbnbv" Jul 09 09:34:18.937095 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:18.937001 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f5bf5aed-640a-478f-a004-94b74b6d3ed9-metrics-tls\") pod \"dns-default-ztphn\" (UID: \"f5bf5aed-640a-478f-a004-94b74b6d3ed9\") " pod="openshift-dns/dns-default-ztphn" Jul 09 09:34:18.937095 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:18.937033 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dffd25e2-f81c-4c5f-afeb-1b56c2930e68-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-456st\" (UID: \"dffd25e2-f81c-4c5f-afeb-1b56c2930e68\") " pod="openshift-insights/insights-runtime-extractor-456st" Jul 09 09:34:18.937095 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:18.937055 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhx47\" (UniqueName: \"kubernetes.io/projected/dffd25e2-f81c-4c5f-afeb-1b56c2930e68-kube-api-access-bhx47\") pod \"insights-runtime-extractor-456st\" (UID: \"dffd25e2-f81c-4c5f-afeb-1b56c2930e68\") " pod="openshift-insights/insights-runtime-extractor-456st" Jul 09 09:34:18.937095 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:18.937082 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/dffd25e2-f81c-4c5f-afeb-1b56c2930e68-crio-socket\") pod \"insights-runtime-extractor-456st\" (UID: \"dffd25e2-f81c-4c5f-afeb-1b56c2930e68\") " pod="openshift-insights/insights-runtime-extractor-456st" Jul 09 09:34:18.937375 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:18.937110 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f5bf5aed-640a-478f-a004-94b74b6d3ed9-config-volume\") pod \"dns-default-ztphn\" (UID: \"f5bf5aed-640a-478f-a004-94b74b6d3ed9\") " pod="openshift-dns/dns-default-ztphn" Jul 09 09:34:18.937375 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:18.937133 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dffd25e2-f81c-4c5f-afeb-1b56c2930e68-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-456st\" (UID: \"dffd25e2-f81c-4c5f-afeb-1b56c2930e68\") " pod="openshift-insights/insights-runtime-extractor-456st" Jul 09 09:34:18.937375 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:18.937174 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/dffd25e2-f81c-4c5f-afeb-1b56c2930e68-data-volume\") pod \"insights-runtime-extractor-456st\" (UID: \"dffd25e2-f81c-4c5f-afeb-1b56c2930e68\") " pod="openshift-insights/insights-runtime-extractor-456st" Jul 09 09:34:18.937375 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:18.937197 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6a11f547-4bad-4cc3-9a49-bc04f9eac0e9-cert\") pod \"ingress-canary-cbnbv\" (UID: \"6a11f547-4bad-4cc3-9a49-bc04f9eac0e9\") " pod="openshift-ingress-canary/ingress-canary-cbnbv" Jul 09 09:34:18.937375 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:18.937290 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/dffd25e2-f81c-4c5f-afeb-1b56c2930e68-crio-socket\") pod \"insights-runtime-extractor-456st\" (UID: \"dffd25e2-f81c-4c5f-afeb-1b56c2930e68\") " pod="openshift-insights/insights-runtime-extractor-456st" Jul 09 09:34:18.937375 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:18.937347 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5bf5aed-640a-478f-a004-94b74b6d3ed9-tmp-dir\") pod \"dns-default-ztphn\" (UID: \"f5bf5aed-640a-478f-a004-94b74b6d3ed9\") " pod="openshift-dns/dns-default-ztphn" Jul 09 09:34:18.937605 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:18.937586 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/dffd25e2-f81c-4c5f-afeb-1b56c2930e68-data-volume\") pod \"insights-runtime-extractor-456st\" (UID: \"dffd25e2-f81c-4c5f-afeb-1b56c2930e68\") " pod="openshift-insights/insights-runtime-extractor-456st" Jul 09 09:34:18.937682 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:18.937655 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f5bf5aed-640a-478f-a004-94b74b6d3ed9-config-volume\") pod \"dns-default-ztphn\" (UID: \"f5bf5aed-640a-478f-a004-94b74b6d3ed9\") " pod="openshift-dns/dns-default-ztphn" Jul 09 09:34:18.937790 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:18.937729 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dffd25e2-f81c-4c5f-afeb-1b56c2930e68-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-456st\" (UID: \"dffd25e2-f81c-4c5f-afeb-1b56c2930e68\") " pod="openshift-insights/insights-runtime-extractor-456st" Jul 09 09:34:18.941157 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:18.941140 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dffd25e2-f81c-4c5f-afeb-1b56c2930e68-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-456st\" (UID: \"dffd25e2-f81c-4c5f-afeb-1b56c2930e68\") " pod="openshift-insights/insights-runtime-extractor-456st" Jul 09 09:34:18.941236 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:18.941183 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f5bf5aed-640a-478f-a004-94b74b6d3ed9-metrics-tls\") pod \"dns-default-ztphn\" (UID: \"f5bf5aed-640a-478f-a004-94b74b6d3ed9\") " pod="openshift-dns/dns-default-ztphn" Jul 09 09:34:18.945643 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:18.945621 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgl9z\" (UniqueName: \"kubernetes.io/projected/f5bf5aed-640a-478f-a004-94b74b6d3ed9-kube-api-access-fgl9z\") pod \"dns-default-ztphn\" (UID: \"f5bf5aed-640a-478f-a004-94b74b6d3ed9\") " pod="openshift-dns/dns-default-ztphn" Jul 09 09:34:18.945643 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:18.945635 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhx47\" (UniqueName: \"kubernetes.io/projected/dffd25e2-f81c-4c5f-afeb-1b56c2930e68-kube-api-access-bhx47\") pod \"insights-runtime-extractor-456st\" (UID: \"dffd25e2-f81c-4c5f-afeb-1b56c2930e68\") " pod="openshift-insights/insights-runtime-extractor-456st" Jul 09 09:34:19.028157 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:19.028108 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ztphn" Jul 09 09:34:19.034252 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:19.034235 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-456st" Jul 09 09:34:19.038503 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:19.038488 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47946\" (UniqueName: \"kubernetes.io/projected/6a11f547-4bad-4cc3-9a49-bc04f9eac0e9-kube-api-access-47946\") pod \"ingress-canary-cbnbv\" (UID: \"6a11f547-4bad-4cc3-9a49-bc04f9eac0e9\") " pod="openshift-ingress-canary/ingress-canary-cbnbv" Jul 09 09:34:19.038552 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:19.038536 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6a11f547-4bad-4cc3-9a49-bc04f9eac0e9-cert\") pod \"ingress-canary-cbnbv\" (UID: \"6a11f547-4bad-4cc3-9a49-bc04f9eac0e9\") " pod="openshift-ingress-canary/ingress-canary-cbnbv" Jul 09 09:34:19.040680 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:19.040658 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6a11f547-4bad-4cc3-9a49-bc04f9eac0e9-cert\") pod \"ingress-canary-cbnbv\" (UID: \"6a11f547-4bad-4cc3-9a49-bc04f9eac0e9\") " pod="openshift-ingress-canary/ingress-canary-cbnbv" Jul 09 09:34:19.047572 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:19.047547 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47946\" (UniqueName: \"kubernetes.io/projected/6a11f547-4bad-4cc3-9a49-bc04f9eac0e9-kube-api-access-47946\") pod \"ingress-canary-cbnbv\" (UID: \"6a11f547-4bad-4cc3-9a49-bc04f9eac0e9\") " pod="openshift-ingress-canary/ingress-canary-cbnbv" Jul 09 09:34:19.059983 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:19.059967 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cbnbv" Jul 09 09:34:19.253080 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:19.253052 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cbnbv"] Jul 09 09:34:19.256890 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:19.256749 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ztphn"] Jul 09 09:34:19.258974 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:34:19.258939 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a11f547_4bad_4cc3_9a49_bc04f9eac0e9.slice/crio-b675eebf298f58ea080d7fdd6f25e023933ccf8a1735a5208cd2c56bfcd0b065 WatchSource:0}: Error finding container b675eebf298f58ea080d7fdd6f25e023933ccf8a1735a5208cd2c56bfcd0b065: Status 404 returned error can't find the container with id b675eebf298f58ea080d7fdd6f25e023933ccf8a1735a5208cd2c56bfcd0b065 Jul 09 09:34:19.260591 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:34:19.260574 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5bf5aed_640a_478f_a004_94b74b6d3ed9.slice/crio-a7e040da5c3c00d54a819967a7c71d2b6623b48049cd14f89c6bfaa52ed38091 WatchSource:0}: Error finding container a7e040da5c3c00d54a819967a7c71d2b6623b48049cd14f89c6bfaa52ed38091: Status 404 returned error can't find the container with id a7e040da5c3c00d54a819967a7c71d2b6623b48049cd14f89c6bfaa52ed38091 Jul 09 09:34:19.286980 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:19.286785 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-456st"] Jul 09 09:34:19.289289 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:34:19.289271 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddffd25e2_f81c_4c5f_afeb_1b56c2930e68.slice/crio-18d8db70d0e661027b0bb5a975e54e240d8832247aae6d05a659c50a7ab6c11e WatchSource:0}: Error finding container 18d8db70d0e661027b0bb5a975e54e240d8832247aae6d05a659c50a7ab6c11e: Status 404 returned error can't find the container with id 18d8db70d0e661027b0bb5a975e54e240d8832247aae6d05a659c50a7ab6c11e Jul 09 09:34:19.438163 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:19.438130 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wsdxs" Jul 09 09:34:19.438964 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:19.438140 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gqlhc" Jul 09 09:34:19.443097 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:19.443078 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 09 09:34:19.443700 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:19.443686 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-btgg6\"" Jul 09 09:34:19.444115 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:19.444093 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 09 09:34:19.444192 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:19.444142 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 09 09:34:19.444192 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:19.444104 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sb7qz\"" Jul 09 09:34:19.691620 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:19.691597 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-dwqgj"] Jul 09 09:34:19.724682 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:19.724651 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-dwqgj"] Jul 09 09:34:19.724780 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:19.724686 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cbnbv" event={"ID":"6a11f547-4bad-4cc3-9a49-bc04f9eac0e9","Type":"ContainerStarted","Data":"b675eebf298f58ea080d7fdd6f25e023933ccf8a1735a5208cd2c56bfcd0b065"} Jul 09 09:34:19.724780 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:19.724711 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ztphn" event={"ID":"f5bf5aed-640a-478f-a004-94b74b6d3ed9","Type":"ContainerStarted","Data":"a7e040da5c3c00d54a819967a7c71d2b6623b48049cd14f89c6bfaa52ed38091"} Jul 09 09:34:19.724780 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:19.724728 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-456st" event={"ID":"dffd25e2-f81c-4c5f-afeb-1b56c2930e68","Type":"ContainerStarted","Data":"18d8db70d0e661027b0bb5a975e54e240d8832247aae6d05a659c50a7ab6c11e"} Jul 09 09:34:19.724780 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:19.724770 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dwqgj" Jul 09 09:34:19.727817 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:19.727797 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 09 09:34:19.727919 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:19.727796 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 09 09:34:19.727968 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:19.727802 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-qnx7v\"" Jul 09 09:34:19.728537 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:19.728522 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 09 09:34:19.728537 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:19.728528 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 09 09:34:19.728642 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:19.728559 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 09 09:34:19.848106 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:19.848076 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1b5a080b-1c43-4958-899a-8fcabb524ba3-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-dwqgj\" (UID: \"1b5a080b-1c43-4958-899a-8fcabb524ba3\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dwqgj" Jul 09 09:34:19.848234 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:19.848126 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1b5a080b-1c43-4958-899a-8fcabb524ba3-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-dwqgj\" (UID: \"1b5a080b-1c43-4958-899a-8fcabb524ba3\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dwqgj" Jul 09 09:34:19.848234 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:19.848198 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b5a080b-1c43-4958-899a-8fcabb524ba3-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-dwqgj\" (UID: \"1b5a080b-1c43-4958-899a-8fcabb524ba3\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dwqgj" Jul 09 09:34:19.848332 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:19.848242 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsx5d\" (UniqueName: \"kubernetes.io/projected/1b5a080b-1c43-4958-899a-8fcabb524ba3-kube-api-access-jsx5d\") pod \"prometheus-operator-567f7b8b7c-dwqgj\" (UID: \"1b5a080b-1c43-4958-899a-8fcabb524ba3\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dwqgj" Jul 09 09:34:19.949614 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:19.949549 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1b5a080b-1c43-4958-899a-8fcabb524ba3-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-dwqgj\" (UID: \"1b5a080b-1c43-4958-899a-8fcabb524ba3\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dwqgj" Jul 09 09:34:19.949614 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:19.949589 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1b5a080b-1c43-4958-899a-8fcabb524ba3-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-dwqgj\" (UID: \"1b5a080b-1c43-4958-899a-8fcabb524ba3\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dwqgj" Jul 09 09:34:19.949788 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:19.949646 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b5a080b-1c43-4958-899a-8fcabb524ba3-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-dwqgj\" (UID: \"1b5a080b-1c43-4958-899a-8fcabb524ba3\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dwqgj" Jul 09 09:34:19.949788 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:19.949673 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsx5d\" (UniqueName: \"kubernetes.io/projected/1b5a080b-1c43-4958-899a-8fcabb524ba3-kube-api-access-jsx5d\") pod \"prometheus-operator-567f7b8b7c-dwqgj\" (UID: \"1b5a080b-1c43-4958-899a-8fcabb524ba3\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dwqgj" Jul 09 09:34:19.951479 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:19.950487 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b5a080b-1c43-4958-899a-8fcabb524ba3-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-dwqgj\" (UID: \"1b5a080b-1c43-4958-899a-8fcabb524ba3\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dwqgj" Jul 09 09:34:19.953285 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:19.953261 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1b5a080b-1c43-4958-899a-8fcabb524ba3-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-dwqgj\" (UID: \"1b5a080b-1c43-4958-899a-8fcabb524ba3\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dwqgj" Jul 09 09:34:19.953407 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:19.953320 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1b5a080b-1c43-4958-899a-8fcabb524ba3-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-dwqgj\" (UID: \"1b5a080b-1c43-4958-899a-8fcabb524ba3\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dwqgj" Jul 09 09:34:19.959361 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:19.959335 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsx5d\" (UniqueName: \"kubernetes.io/projected/1b5a080b-1c43-4958-899a-8fcabb524ba3-kube-api-access-jsx5d\") pod \"prometheus-operator-567f7b8b7c-dwqgj\" (UID: \"1b5a080b-1c43-4958-899a-8fcabb524ba3\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dwqgj" Jul 09 09:34:20.033245 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:20.033217 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dwqgj" Jul 09 09:34:20.205942 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:20.205867 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-dwqgj"] Jul 09 09:34:20.210519 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:34:20.210489 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b5a080b_1c43_4958_899a_8fcabb524ba3.slice/crio-be1500e691738e6514efecb24050d8b2734a5125cf44417c5691f645c9399843 WatchSource:0}: Error finding container be1500e691738e6514efecb24050d8b2734a5125cf44417c5691f645c9399843: Status 404 returned error can't find the container with id be1500e691738e6514efecb24050d8b2734a5125cf44417c5691f645c9399843 Jul 09 09:34:20.438585 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:20.437987 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sw9zq" Jul 09 09:34:20.441202 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:20.441008 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 09 09:34:20.727152 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:20.727079 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dwqgj" event={"ID":"1b5a080b-1c43-4958-899a-8fcabb524ba3","Type":"ContainerStarted","Data":"be1500e691738e6514efecb24050d8b2734a5125cf44417c5691f645c9399843"} Jul 09 09:34:26.745051 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:26.744863 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-456st" event={"ID":"dffd25e2-f81c-4c5f-afeb-1b56c2930e68","Type":"ContainerStarted","Data":"3e4a3c8cbdde9bcfdb6edd1c0068857fc61046083f4d612cc401a232ae5fbceb"} Jul 09 09:34:26.749621 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:26.749593 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dwqgj" event={"ID":"1b5a080b-1c43-4958-899a-8fcabb524ba3","Type":"ContainerStarted","Data":"101408f4acde6fb708e72bebe986d770ac90e9325639f29dddbcfa41b7ebcbee"} Jul 09 09:34:26.749768 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:26.749675 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dwqgj" event={"ID":"1b5a080b-1c43-4958-899a-8fcabb524ba3","Type":"ContainerStarted","Data":"da375ebb9460234d4518c1870189d7a01099bd526946fa129c244d9d55e644a8"} Jul 09 09:34:26.751353 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:26.751332 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ztphn" event={"ID":"f5bf5aed-640a-478f-a004-94b74b6d3ed9","Type":"ContainerStarted","Data":"ba0afe164ad521bcbb3a76a3df7b2d1f9cf72926d5454fd3c0042194487d486f"} Jul 09 09:34:26.751460 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:26.751366 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ztphn" event={"ID":"f5bf5aed-640a-478f-a004-94b74b6d3ed9","Type":"ContainerStarted","Data":"488822401b8a6c201b0532d2d5a209fa144dfab5d521b36781dee5a99b41a771"} Jul 09 09:34:26.751460 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:26.751386 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-ztphn" Jul 09 09:34:26.752621 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:26.752604 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cbnbv" event={"ID":"6a11f547-4bad-4cc3-9a49-bc04f9eac0e9","Type":"ContainerStarted","Data":"10c2369c4b7805a18cd379cf7f667ac898be2362f5bb5216ece48215983d892e"} Jul 09 09:34:26.784994 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:26.784960 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dwqgj" podStartSLOduration=1.888122093 podStartE2EDuration="7.784949279s" totalImagesPullingTime="5.896827186s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:34:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:34:26.78381016 +0000 UTC m=+51.875564080" watchObservedRunningTime="2026-07-09 09:34:26.784949279 +0000 UTC m=+51.876703198" Jul 09 09:34:26.814214 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:26.814176 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-cbnbv" podStartSLOduration=1.96295075 podStartE2EDuration="8.814164473s" totalImagesPullingTime="6.851213723s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:34:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:34:26.812684965 +0000 UTC m=+51.904438876" watchObservedRunningTime="2026-07-09 09:34:26.814164473 +0000 UTC m=+51.905918371" Jul 09 09:34:26.843946 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:26.843907 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-ztphn" podStartSLOduration=1.994806651 podStartE2EDuration="8.843897661s" totalImagesPullingTime="6.84909101s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:34:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:34:26.842432667 +0000 UTC m=+51.934186587" watchObservedRunningTime="2026-07-09 09:34:26.843897661 +0000 UTC m=+51.935651582" Jul 09 09:34:27.757342 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:27.757309 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-456st" event={"ID":"dffd25e2-f81c-4c5f-afeb-1b56c2930e68","Type":"ContainerStarted","Data":"d5900c3a45d161eb7a6d9d0d7893facb1e6a93a1856c610eddeae09b08a8a080"} Jul 09 09:34:28.391103 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:28.391061 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-685bfb5f64-9hk94"] Jul 09 09:34:28.392871 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:28.392833 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-9hk94" Jul 09 09:34:28.395454 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:28.395248 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-gj84c\"" Jul 09 09:34:28.395635 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:28.395282 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 09 09:34:28.395635 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:28.395282 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 09 09:34:28.404385 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:28.404364 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-9hk94"] Jul 09 09:34:28.413634 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:28.413610 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvdfz\" (UniqueName: \"kubernetes.io/projected/18bcc96c-0950-49b5-b109-fdd52cfd0616-kube-api-access-qvdfz\") pod \"downloads-685bfb5f64-9hk94\" (UID: \"18bcc96c-0950-49b5-b109-fdd52cfd0616\") " pod="openshift-console/downloads-685bfb5f64-9hk94" Jul 09 09:34:28.514287 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:28.514256 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvdfz\" (UniqueName: \"kubernetes.io/projected/18bcc96c-0950-49b5-b109-fdd52cfd0616-kube-api-access-qvdfz\") pod \"downloads-685bfb5f64-9hk94\" (UID: \"18bcc96c-0950-49b5-b109-fdd52cfd0616\") " pod="openshift-console/downloads-685bfb5f64-9hk94" Jul 09 09:34:28.525257 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:28.525230 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvdfz\" (UniqueName: \"kubernetes.io/projected/18bcc96c-0950-49b5-b109-fdd52cfd0616-kube-api-access-qvdfz\") pod \"downloads-685bfb5f64-9hk94\" (UID: \"18bcc96c-0950-49b5-b109-fdd52cfd0616\") " pod="openshift-console/downloads-685bfb5f64-9hk94" Jul 09 09:34:28.703499 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:28.703386 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-9hk94" Jul 09 09:34:28.864802 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:28.864769 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-9hk94"] Jul 09 09:34:28.868591 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:34:28.868559 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18bcc96c_0950_49b5_b109_fdd52cfd0616.slice/crio-cebb04e430de634d4f861b7fa09160903145d6408dc4f4e4968f54fa90e54f6c WatchSource:0}: Error finding container cebb04e430de634d4f861b7fa09160903145d6408dc4f4e4968f54fa90e54f6c: Status 404 returned error can't find the container with id cebb04e430de634d4f861b7fa09160903145d6408dc4f4e4968f54fa90e54f6c Jul 09 09:34:29.116981 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:29.116913 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-q9lb8"] Jul 09 09:34:29.120826 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:29.120805 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-q9lb8" Jul 09 09:34:29.124912 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:29.124527 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 09 09:34:29.124912 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:29.124566 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-fbsl6\"" Jul 09 09:34:29.124912 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:29.124530 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 09 09:34:29.125126 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:29.124953 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 09 09:34:29.139392 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:29.139356 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-q9lb8"] Jul 09 09:34:29.140308 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:29.140288 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-s2dl4"] Jul 09 09:34:29.142498 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:29.142478 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-s2dl4" Jul 09 09:34:29.145490 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:29.145469 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 09 09:34:29.145717 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:29.145702 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-jsrjg\"" Jul 09 09:34:29.145900 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:29.145889 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 09 09:34:29.146313 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:29.146296 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 09 09:34:29.221343 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:29.221311 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6b5a52bf-1368-4232-a345-09f786e28558-node-exporter-textfile\") pod \"node-exporter-s2dl4\" (UID: \"6b5a52bf-1368-4232-a345-09f786e28558\") " pod="openshift-monitoring/node-exporter-s2dl4" Jul 09 09:34:29.221489 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:29.221355 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6b5a52bf-1368-4232-a345-09f786e28558-node-exporter-tls\") pod \"node-exporter-s2dl4\" (UID: \"6b5a52bf-1368-4232-a345-09f786e28558\") " pod="openshift-monitoring/node-exporter-s2dl4" Jul 09 09:34:29.221489 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:29.221392 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6b5a52bf-1368-4232-a345-09f786e28558-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s2dl4\" (UID: \"6b5a52bf-1368-4232-a345-09f786e28558\") " pod="openshift-monitoring/node-exporter-s2dl4" Jul 09 09:34:29.221489 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:29.221427 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e4b1e242-011d-4b72-8aaa-1e6d75842178-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-q9lb8\" (UID: \"e4b1e242-011d-4b72-8aaa-1e6d75842178\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-q9lb8" Jul 09 09:34:29.221489 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:29.221449 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e4b1e242-011d-4b72-8aaa-1e6d75842178-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-q9lb8\" (UID: \"e4b1e242-011d-4b72-8aaa-1e6d75842178\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-q9lb8" Jul 09 09:34:29.221489 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:29.221474 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhh6j\" (UniqueName: \"kubernetes.io/projected/e4b1e242-011d-4b72-8aaa-1e6d75842178-kube-api-access-hhh6j\") pod \"kube-state-metrics-7c9dbb8b7d-q9lb8\" (UID: \"e4b1e242-011d-4b72-8aaa-1e6d75842178\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-q9lb8" Jul 09 09:34:29.221691 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:29.221495 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4b1e242-011d-4b72-8aaa-1e6d75842178-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-q9lb8\" (UID: \"e4b1e242-011d-4b72-8aaa-1e6d75842178\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-q9lb8" Jul 09 09:34:29.221691 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:29.221530 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e4b1e242-011d-4b72-8aaa-1e6d75842178-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-q9lb8\" (UID: \"e4b1e242-011d-4b72-8aaa-1e6d75842178\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-q9lb8" Jul 09 09:34:29.221691 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:29.221552 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6b5a52bf-1368-4232-a345-09f786e28558-node-exporter-wtmp\") pod \"node-exporter-s2dl4\" (UID: \"6b5a52bf-1368-4232-a345-09f786e28558\") " pod="openshift-monitoring/node-exporter-s2dl4" Jul 09 09:34:29.221691 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:29.221589 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plwpl\" (UniqueName: \"kubernetes.io/projected/6b5a52bf-1368-4232-a345-09f786e28558-kube-api-access-plwpl\") pod \"node-exporter-s2dl4\" (UID: \"6b5a52bf-1368-4232-a345-09f786e28558\") " pod="openshift-monitoring/node-exporter-s2dl4" Jul 09 09:34:29.221691 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:29.221615 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6b5a52bf-1368-4232-a345-09f786e28558-root\") pod \"node-exporter-s2dl4\" (UID: \"6b5a52bf-1368-4232-a345-09f786e28558\") " pod="openshift-monitoring/node-exporter-s2dl4" Jul 09 09:34:29.221691 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:29.221650 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6b5a52bf-1368-4232-a345-09f786e28558-node-exporter-accelerators-collector-config\") pod \"node-exporter-s2dl4\" (UID: \"6b5a52bf-1368-4232-a345-09f786e28558\") " pod="openshift-monitoring/node-exporter-s2dl4" Jul 09 09:34:29.221691 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:29.221676 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b5a52bf-1368-4232-a345-09f786e28558-sys\") pod \"node-exporter-s2dl4\" (UID: \"6b5a52bf-1368-4232-a345-09f786e28558\") " pod="openshift-monitoring/node-exporter-s2dl4" Jul 09 09:34:29.221988 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:29.221709 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e4b1e242-011d-4b72-8aaa-1e6d75842178-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-q9lb8\" (UID: \"e4b1e242-011d-4b72-8aaa-1e6d75842178\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-q9lb8" Jul 09 09:34:29.221988 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:29.221726 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b5a52bf-1368-4232-a345-09f786e28558-metrics-client-ca\") pod \"node-exporter-s2dl4\" (UID: \"6b5a52bf-1368-4232-a345-09f786e28558\") " pod="openshift-monitoring/node-exporter-s2dl4" Jul 09 09:34:29.322791 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:29.322753 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6b5a52bf-1368-4232-a345-09f786e28558-node-exporter-accelerators-collector-config\") pod \"node-exporter-s2dl4\" (UID: \"6b5a52bf-1368-4232-a345-09f786e28558\") " pod="openshift-monitoring/node-exporter-s2dl4" Jul 09 09:34:29.322958 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:29.322801 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b5a52bf-1368-4232-a345-09f786e28558-sys\") pod \"node-exporter-s2dl4\" (UID: \"6b5a52bf-1368-4232-a345-09f786e28558\") " pod="openshift-monitoring/node-exporter-s2dl4" Jul 09 09:34:29.322958 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:29.322865 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e4b1e242-011d-4b72-8aaa-1e6d75842178-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-q9lb8\" (UID: \"e4b1e242-011d-4b72-8aaa-1e6d75842178\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-q9lb8" Jul 09 09:34:29.322958 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:29.322893 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b5a52bf-1368-4232-a345-09f786e28558-metrics-client-ca\") pod \"node-exporter-s2dl4\" (UID: \"6b5a52bf-1368-4232-a345-09f786e28558\") " pod="openshift-monitoring/node-exporter-s2dl4" Jul 09 09:34:29.322958 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:29.322920 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b5a52bf-1368-4232-a345-09f786e28558-sys\") pod \"node-exporter-s2dl4\" (UID: \"6b5a52bf-1368-4232-a345-09f786e28558\") " pod="openshift-monitoring/node-exporter-s2dl4" Jul 09 09:34:29.322958 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:29.322924 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6b5a52bf-1368-4232-a345-09f786e28558-node-exporter-textfile\") pod \"node-exporter-s2dl4\" (UID: \"6b5a52bf-1368-4232-a345-09f786e28558\") " pod="openshift-monitoring/node-exporter-s2dl4" Jul 09 09:34:29.323151 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:29.322990 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6b5a52bf-1368-4232-a345-09f786e28558-node-exporter-tls\") pod \"node-exporter-s2dl4\" (UID: \"6b5a52bf-1368-4232-a345-09f786e28558\") " pod="openshift-monitoring/node-exporter-s2dl4" Jul 09 09:34:29.323151 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:29.323028 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6b5a52bf-1368-4232-a345-09f786e28558-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s2dl4\" (UID: \"6b5a52bf-1368-4232-a345-09f786e28558\") " pod="openshift-monitoring/node-exporter-s2dl4" Jul 09 09:34:29.323151 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:29.323069 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e4b1e242-011d-4b72-8aaa-1e6d75842178-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-q9lb8\" (UID: \"e4b1e242-011d-4b72-8aaa-1e6d75842178\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-q9lb8" Jul 09 09:34:29.323151 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:29.323097 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e4b1e242-011d-4b72-8aaa-1e6d75842178-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-q9lb8\" (UID: \"e4b1e242-011d-4b72-8aaa-1e6d75842178\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-q9lb8" Jul 09 09:34:29.323151 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:29.323120 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhh6j\" (UniqueName: \"kubernetes.io/projected/e4b1e242-011d-4b72-8aaa-1e6d75842178-kube-api-access-hhh6j\") pod \"kube-state-metrics-7c9dbb8b7d-q9lb8\" (UID: \"e4b1e242-011d-4b72-8aaa-1e6d75842178\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-q9lb8" Jul 09 09:34:29.323151 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:29.323145 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4b1e242-011d-4b72-8aaa-1e6d75842178-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-q9lb8\" (UID: \"e4b1e242-011d-4b72-8aaa-1e6d75842178\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-q9lb8" Jul 09 09:34:29.323420 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:29.323180 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e4b1e242-011d-4b72-8aaa-1e6d75842178-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-q9lb8\" (UID: \"e4b1e242-011d-4b72-8aaa-1e6d75842178\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-q9lb8" Jul 09 09:34:29.323420 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:29.323205 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6b5a52bf-1368-4232-a345-09f786e28558-node-exporter-wtmp\") pod \"node-exporter-s2dl4\" (UID: \"6b5a52bf-1368-4232-a345-09f786e28558\") " pod="openshift-monitoring/node-exporter-s2dl4" Jul 09 09:34:29.323420 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:34:29.323240 2578 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 09 09:34:29.323420 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:29.323253 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plwpl\" (UniqueName: \"kubernetes.io/projected/6b5a52bf-1368-4232-a345-09f786e28558-kube-api-access-plwpl\") pod \"node-exporter-s2dl4\" (UID: \"6b5a52bf-1368-4232-a345-09f786e28558\") " pod="openshift-monitoring/node-exporter-s2dl4" Jul 09 09:34:29.323420 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:29.323283 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6b5a52bf-1368-4232-a345-09f786e28558-root\") pod \"node-exporter-s2dl4\" (UID: \"6b5a52bf-1368-4232-a345-09f786e28558\") " pod="openshift-monitoring/node-exporter-s2dl4" Jul 09 09:34:29.323420 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:29.323299 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e4b1e242-011d-4b72-8aaa-1e6d75842178-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-q9lb8\" (UID: \"e4b1e242-011d-4b72-8aaa-1e6d75842178\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-q9lb8" Jul 09 09:34:29.323420 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:34:29.323310 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b5a52bf-1368-4232-a345-09f786e28558-node-exporter-tls podName:6b5a52bf-1368-4232-a345-09f786e28558 nodeName:}" failed. No retries permitted until 2026-07-09 09:34:29.823290226 +0000 UTC m=+54.915044152 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/6b5a52bf-1368-4232-a345-09f786e28558-node-exporter-tls") pod "node-exporter-s2dl4" (UID: "6b5a52bf-1368-4232-a345-09f786e28558") : secret "node-exporter-tls" not found Jul 09 09:34:29.323420 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:29.323325 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6b5a52bf-1368-4232-a345-09f786e28558-root\") pod \"node-exporter-s2dl4\" (UID: \"6b5a52bf-1368-4232-a345-09f786e28558\") " pod="openshift-monitoring/node-exporter-s2dl4" Jul 09 09:34:29.323420 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:29.323411 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6b5a52bf-1368-4232-a345-09f786e28558-node-exporter-accelerators-collector-config\") pod \"node-exporter-s2dl4\" (UID: \"6b5a52bf-1368-4232-a345-09f786e28558\") " pod="openshift-monitoring/node-exporter-s2dl4" Jul 09 09:34:29.323873 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:29.323535 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6b5a52bf-1368-4232-a345-09f786e28558-node-exporter-wtmp\") pod \"node-exporter-s2dl4\" (UID: \"6b5a52bf-1368-4232-a345-09f786e28558\") " pod="openshift-monitoring/node-exporter-s2dl4" Jul 09 09:34:29.323873 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:34:29.323609 2578 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 09 09:34:29.323873 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:34:29.323709 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4b1e242-011d-4b72-8aaa-1e6d75842178-kube-state-metrics-tls podName:e4b1e242-011d-4b72-8aaa-1e6d75842178 nodeName:}" failed. No retries permitted until 2026-07-09 09:34:29.823696829 +0000 UTC m=+54.915450728 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/e4b1e242-011d-4b72-8aaa-1e6d75842178-kube-state-metrics-tls") pod "kube-state-metrics-7c9dbb8b7d-q9lb8" (UID: "e4b1e242-011d-4b72-8aaa-1e6d75842178") : secret "kube-state-metrics-tls" not found Jul 09 09:34:29.323873 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:29.323855 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6b5a52bf-1368-4232-a345-09f786e28558-node-exporter-textfile\") pod \"node-exporter-s2dl4\" (UID: \"6b5a52bf-1368-4232-a345-09f786e28558\") " pod="openshift-monitoring/node-exporter-s2dl4" Jul 09 09:34:29.324079 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:29.324061 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e4b1e242-011d-4b72-8aaa-1e6d75842178-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-q9lb8\" (UID: \"e4b1e242-011d-4b72-8aaa-1e6d75842178\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-q9lb8" Jul 09 09:34:29.324175 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:29.324153 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b5a52bf-1368-4232-a345-09f786e28558-metrics-client-ca\") pod \"node-exporter-s2dl4\" (UID: \"6b5a52bf-1368-4232-a345-09f786e28558\") " pod="openshift-monitoring/node-exporter-s2dl4" Jul 09 09:34:29.324248 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:29.324228 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4b1e242-011d-4b72-8aaa-1e6d75842178-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-q9lb8\" (UID: \"e4b1e242-011d-4b72-8aaa-1e6d75842178\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-q9lb8" Jul 09 09:34:29.327174 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:29.327150 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6b5a52bf-1368-4232-a345-09f786e28558-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s2dl4\" (UID: \"6b5a52bf-1368-4232-a345-09f786e28558\") " pod="openshift-monitoring/node-exporter-s2dl4" Jul 09 09:34:29.327297 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:29.327262 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e4b1e242-011d-4b72-8aaa-1e6d75842178-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-q9lb8\" (UID: \"e4b1e242-011d-4b72-8aaa-1e6d75842178\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-q9lb8" Jul 09 09:34:29.332871 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:29.332808 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhh6j\" (UniqueName: \"kubernetes.io/projected/e4b1e242-011d-4b72-8aaa-1e6d75842178-kube-api-access-hhh6j\") pod \"kube-state-metrics-7c9dbb8b7d-q9lb8\" (UID: \"e4b1e242-011d-4b72-8aaa-1e6d75842178\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-q9lb8" Jul 09 09:34:29.333471 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:29.333452 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plwpl\" (UniqueName: \"kubernetes.io/projected/6b5a52bf-1368-4232-a345-09f786e28558-kube-api-access-plwpl\") pod \"node-exporter-s2dl4\" (UID: \"6b5a52bf-1368-4232-a345-09f786e28558\") " pod="openshift-monitoring/node-exporter-s2dl4" Jul 09 09:34:29.766826 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:29.766786 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-9hk94" event={"ID":"18bcc96c-0950-49b5-b109-fdd52cfd0616","Type":"ContainerStarted","Data":"cebb04e430de634d4f861b7fa09160903145d6408dc4f4e4968f54fa90e54f6c"} Jul 09 09:34:29.828171 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:29.828118 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6b5a52bf-1368-4232-a345-09f786e28558-node-exporter-tls\") pod \"node-exporter-s2dl4\" (UID: \"6b5a52bf-1368-4232-a345-09f786e28558\") " pod="openshift-monitoring/node-exporter-s2dl4" Jul 09 09:34:29.828335 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:29.828199 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e4b1e242-011d-4b72-8aaa-1e6d75842178-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-q9lb8\" (UID: \"e4b1e242-011d-4b72-8aaa-1e6d75842178\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-q9lb8" Jul 09 09:34:29.830803 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:29.830777 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6b5a52bf-1368-4232-a345-09f786e28558-node-exporter-tls\") pod \"node-exporter-s2dl4\" (UID: \"6b5a52bf-1368-4232-a345-09f786e28558\") " pod="openshift-monitoring/node-exporter-s2dl4" Jul 09 09:34:29.832085 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:29.832062 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e4b1e242-011d-4b72-8aaa-1e6d75842178-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-q9lb8\" (UID: \"e4b1e242-011d-4b72-8aaa-1e6d75842178\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-q9lb8" Jul 09 09:34:30.033789 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:30.033718 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-q9lb8" Jul 09 09:34:30.054006 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:30.053983 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-s2dl4" Jul 09 09:34:30.255347 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:34:30.255314 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b5a52bf_1368_4232_a345_09f786e28558.slice/crio-8df491c8ff1b92cfd5e6ad8c93e99adb298de33a0c784aab5b560dc82627dc64 WatchSource:0}: Error finding container 8df491c8ff1b92cfd5e6ad8c93e99adb298de33a0c784aab5b560dc82627dc64: Status 404 returned error can't find the container with id 8df491c8ff1b92cfd5e6ad8c93e99adb298de33a0c784aab5b560dc82627dc64 Jul 09 09:34:30.398598 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:30.398362 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-q9lb8"] Jul 09 09:34:30.404420 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:34:30.404392 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4b1e242_011d_4b72_8aaa_1e6d75842178.slice/crio-c6345c48901afdb892bd0dc632fd8ac4f6580bde06a0abd12e4b60c4e29fa7d3 WatchSource:0}: Error finding container c6345c48901afdb892bd0dc632fd8ac4f6580bde06a0abd12e4b60c4e29fa7d3: Status 404 returned error can't find the container with id c6345c48901afdb892bd0dc632fd8ac4f6580bde06a0abd12e4b60c4e29fa7d3 Jul 09 09:34:30.771863 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:30.771810 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-q9lb8" event={"ID":"e4b1e242-011d-4b72-8aaa-1e6d75842178","Type":"ContainerStarted","Data":"c6345c48901afdb892bd0dc632fd8ac4f6580bde06a0abd12e4b60c4e29fa7d3"} Jul 09 09:34:30.774137 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:30.774109 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-456st" event={"ID":"dffd25e2-f81c-4c5f-afeb-1b56c2930e68","Type":"ContainerStarted","Data":"da55e069428653c20a20ab18ea4c3a3fd603384dcabe0785038f926f42b67335"} Jul 09 09:34:30.775757 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:30.775735 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s2dl4" event={"ID":"6b5a52bf-1368-4232-a345-09f786e28558","Type":"ContainerStarted","Data":"8df491c8ff1b92cfd5e6ad8c93e99adb298de33a0c784aab5b560dc82627dc64"} Jul 09 09:34:30.793806 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:30.793764 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-456st" podStartSLOduration=2.060017773 podStartE2EDuration="12.79374893s" totalImagesPullingTime="10.733731157s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 09:34:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:34:30.793327693 +0000 UTC m=+55.885081614" watchObservedRunningTime="2026-07-09 09:34:30.79374893 +0000 UTC m=+55.885502851" Jul 09 09:34:31.781168 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:31.781130 2578 generic.go:356] "Generic (PLEG): container finished" podID="6b5a52bf-1368-4232-a345-09f786e28558" containerID="d4a2b3bafa8979f4308b1494569034388abb76a69efb6727daa1c9cef91e2d37" exitCode=0 Jul 09 09:34:31.781628 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:31.781220 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s2dl4" event={"ID":"6b5a52bf-1368-4232-a345-09f786e28558","Type":"ContainerDied","Data":"d4a2b3bafa8979f4308b1494569034388abb76a69efb6727daa1c9cef91e2d37"} Jul 09 09:34:32.787063 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:32.787021 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-q9lb8" event={"ID":"e4b1e242-011d-4b72-8aaa-1e6d75842178","Type":"ContainerStarted","Data":"0ce5e128586e3c760850b3a16c5e0318047647c22d87f7b00ee062d1a5de1f81"} Jul 09 09:34:32.787492 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:32.787131 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-q9lb8" event={"ID":"e4b1e242-011d-4b72-8aaa-1e6d75842178","Type":"ContainerStarted","Data":"3ec4bab7b55e690201ed2be5fc9d5e4e5ad66a0cc9ea8dc65aac70baa0951705"} Jul 09 09:34:32.787492 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:32.787161 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-q9lb8" event={"ID":"e4b1e242-011d-4b72-8aaa-1e6d75842178","Type":"ContainerStarted","Data":"5ef86703a0fd5a10212c9ea1f8776794a8b9e0b4b8be8aa0469766b320506c3f"} Jul 09 09:34:32.789364 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:32.789338 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s2dl4" event={"ID":"6b5a52bf-1368-4232-a345-09f786e28558","Type":"ContainerStarted","Data":"0e843c1f8ed622261e20ee8855c79d19ec7b80e6ae774a7b6f10206710a372b7"} Jul 09 09:34:32.789491 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:32.789384 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s2dl4" event={"ID":"6b5a52bf-1368-4232-a345-09f786e28558","Type":"ContainerStarted","Data":"085fd8db31c01249a47e5e79bc3273ecb1b465d922169a8cc48cd4d0de0e1f3f"} Jul 09 09:34:32.827385 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:32.827333 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-q9lb8" podStartSLOduration=2.229087273 podStartE2EDuration="3.827313885s" totalImagesPullingTime="1.598226612s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:34:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:34:32.808062692 +0000 UTC m=+57.899816612" watchObservedRunningTime="2026-07-09 09:34:32.827313885 +0000 UTC m=+57.919067808" Jul 09 09:34:32.827560 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:32.827471 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-s2dl4" podStartSLOduration=2.652573636 podStartE2EDuration="3.827463529s" totalImagesPullingTime="1.090517519s" totalInitContainerRuntime="84.372374ms" isStatefulPod=true podCreationTimestamp="2026-07-09 09:34:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:34:32.826944848 +0000 UTC m=+57.918698768" watchObservedRunningTime="2026-07-09 09:34:32.827463529 +0000 UTC m=+57.919217450" Jul 09 09:34:33.323131 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:33.323095 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-88f9f95-bfngx"] Jul 09 09:34:33.327714 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:33.327690 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-88f9f95-bfngx" Jul 09 09:34:33.331301 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:33.331278 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-5sxl6\"" Jul 09 09:34:33.331419 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:33.331350 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 09 09:34:33.331419 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:33.331365 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 09 09:34:33.331499 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:33.331461 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 09 09:34:33.331534 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:33.331522 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 09 09:34:33.335506 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:33.335461 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 09 09:34:33.337924 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:33.337900 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-88f9f95-bfngx"] Jul 09 09:34:33.357716 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:33.357674 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c20bfb7c-f525-481e-89c3-2852fce4ffb3-console-serving-cert\") pod \"console-88f9f95-bfngx\" (UID: \"c20bfb7c-f525-481e-89c3-2852fce4ffb3\") " pod="openshift-console/console-88f9f95-bfngx" Jul 09 09:34:33.357908 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:33.357751 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c20bfb7c-f525-481e-89c3-2852fce4ffb3-console-oauth-config\") pod \"console-88f9f95-bfngx\" (UID: \"c20bfb7c-f525-481e-89c3-2852fce4ffb3\") " pod="openshift-console/console-88f9f95-bfngx" Jul 09 09:34:33.357908 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:33.357783 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c20bfb7c-f525-481e-89c3-2852fce4ffb3-oauth-serving-cert\") pod \"console-88f9f95-bfngx\" (UID: \"c20bfb7c-f525-481e-89c3-2852fce4ffb3\") " pod="openshift-console/console-88f9f95-bfngx" Jul 09 09:34:33.357908 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:33.357876 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c20bfb7c-f525-481e-89c3-2852fce4ffb3-service-ca\") pod \"console-88f9f95-bfngx\" (UID: \"c20bfb7c-f525-481e-89c3-2852fce4ffb3\") " pod="openshift-console/console-88f9f95-bfngx" Jul 09 09:34:33.357908 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:33.357907 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvhb4\" (UniqueName: \"kubernetes.io/projected/c20bfb7c-f525-481e-89c3-2852fce4ffb3-kube-api-access-gvhb4\") pod \"console-88f9f95-bfngx\" (UID: \"c20bfb7c-f525-481e-89c3-2852fce4ffb3\") " pod="openshift-console/console-88f9f95-bfngx" Jul 09 09:34:33.358109 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:33.357936 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c20bfb7c-f525-481e-89c3-2852fce4ffb3-console-config\") pod \"console-88f9f95-bfngx\" (UID: \"c20bfb7c-f525-481e-89c3-2852fce4ffb3\") " pod="openshift-console/console-88f9f95-bfngx" Jul 09 09:34:33.458587 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:33.458557 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c20bfb7c-f525-481e-89c3-2852fce4ffb3-console-oauth-config\") pod \"console-88f9f95-bfngx\" (UID: \"c20bfb7c-f525-481e-89c3-2852fce4ffb3\") " pod="openshift-console/console-88f9f95-bfngx" Jul 09 09:34:33.458774 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:33.458607 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c20bfb7c-f525-481e-89c3-2852fce4ffb3-oauth-serving-cert\") pod \"console-88f9f95-bfngx\" (UID: \"c20bfb7c-f525-481e-89c3-2852fce4ffb3\") " pod="openshift-console/console-88f9f95-bfngx" Jul 09 09:34:33.458774 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:33.458660 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c20bfb7c-f525-481e-89c3-2852fce4ffb3-service-ca\") pod \"console-88f9f95-bfngx\" (UID: \"c20bfb7c-f525-481e-89c3-2852fce4ffb3\") " pod="openshift-console/console-88f9f95-bfngx" Jul 09 09:34:33.458774 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:33.458690 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvhb4\" (UniqueName: \"kubernetes.io/projected/c20bfb7c-f525-481e-89c3-2852fce4ffb3-kube-api-access-gvhb4\") pod \"console-88f9f95-bfngx\" (UID: \"c20bfb7c-f525-481e-89c3-2852fce4ffb3\") " pod="openshift-console/console-88f9f95-bfngx" Jul 09 09:34:33.458774 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:33.458727 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c20bfb7c-f525-481e-89c3-2852fce4ffb3-console-config\") pod \"console-88f9f95-bfngx\" (UID: \"c20bfb7c-f525-481e-89c3-2852fce4ffb3\") " pod="openshift-console/console-88f9f95-bfngx" Jul 09 09:34:33.458975 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:33.458781 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c20bfb7c-f525-481e-89c3-2852fce4ffb3-console-serving-cert\") pod \"console-88f9f95-bfngx\" (UID: \"c20bfb7c-f525-481e-89c3-2852fce4ffb3\") " pod="openshift-console/console-88f9f95-bfngx" Jul 09 09:34:33.459570 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:33.459546 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c20bfb7c-f525-481e-89c3-2852fce4ffb3-service-ca\") pod \"console-88f9f95-bfngx\" (UID: \"c20bfb7c-f525-481e-89c3-2852fce4ffb3\") " pod="openshift-console/console-88f9f95-bfngx" Jul 09 09:34:33.459665 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:33.459543 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c20bfb7c-f525-481e-89c3-2852fce4ffb3-console-config\") pod \"console-88f9f95-bfngx\" (UID: \"c20bfb7c-f525-481e-89c3-2852fce4ffb3\") " pod="openshift-console/console-88f9f95-bfngx" Jul 09 09:34:33.459796 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:33.459751 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c20bfb7c-f525-481e-89c3-2852fce4ffb3-oauth-serving-cert\") pod \"console-88f9f95-bfngx\" (UID: \"c20bfb7c-f525-481e-89c3-2852fce4ffb3\") " pod="openshift-console/console-88f9f95-bfngx" Jul 09 09:34:33.462081 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:33.462062 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c20bfb7c-f525-481e-89c3-2852fce4ffb3-console-oauth-config\") pod \"console-88f9f95-bfngx\" (UID: \"c20bfb7c-f525-481e-89c3-2852fce4ffb3\") " pod="openshift-console/console-88f9f95-bfngx" Jul 09 09:34:33.462272 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:33.462243 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c20bfb7c-f525-481e-89c3-2852fce4ffb3-console-serving-cert\") pod \"console-88f9f95-bfngx\" (UID: \"c20bfb7c-f525-481e-89c3-2852fce4ffb3\") " pod="openshift-console/console-88f9f95-bfngx" Jul 09 09:34:33.470059 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:33.470033 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvhb4\" (UniqueName: \"kubernetes.io/projected/c20bfb7c-f525-481e-89c3-2852fce4ffb3-kube-api-access-gvhb4\") pod \"console-88f9f95-bfngx\" (UID: \"c20bfb7c-f525-481e-89c3-2852fce4ffb3\") " pod="openshift-console/console-88f9f95-bfngx" Jul 09 09:34:33.639397 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:33.639358 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-88f9f95-bfngx" Jul 09 09:34:33.819888 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:33.819831 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-88f9f95-bfngx"] Jul 09 09:34:33.823509 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:34:33.823479 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc20bfb7c_f525_481e_89c3_2852fce4ffb3.slice/crio-90824acbb52a83a9196c98120631c4ba854377876c96c709914e578f80443d0f WatchSource:0}: Error finding container 90824acbb52a83a9196c98120631c4ba854377876c96c709914e578f80443d0f: Status 404 returned error can't find the container with id 90824acbb52a83a9196c98120631c4ba854377876c96c709914e578f80443d0f Jul 09 09:34:33.854551 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:33.854521 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-hgks7"] Jul 09 09:34:33.886243 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:33.886212 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-hgks7"] Jul 09 09:34:33.886413 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:33.886353 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-hgks7" Jul 09 09:34:33.888940 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:33.888882 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 09 09:34:33.889312 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:33.889291 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-r6npm\"" Jul 09 09:34:33.964644 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:33.964609 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmbc4\" (UniqueName: \"kubernetes.io/projected/675a460e-eff5-4cc7-b339-d104c9bbe310-kube-api-access-jmbc4\") pod \"monitoring-plugin-d5dcff7cf-hgks7\" (UID: \"675a460e-eff5-4cc7-b339-d104c9bbe310\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-hgks7" Jul 09 09:34:33.964811 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:33.964663 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/675a460e-eff5-4cc7-b339-d104c9bbe310-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-hgks7\" (UID: \"675a460e-eff5-4cc7-b339-d104c9bbe310\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-hgks7" Jul 09 09:34:34.066017 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:34.065978 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmbc4\" (UniqueName: \"kubernetes.io/projected/675a460e-eff5-4cc7-b339-d104c9bbe310-kube-api-access-jmbc4\") pod \"monitoring-plugin-d5dcff7cf-hgks7\" (UID: \"675a460e-eff5-4cc7-b339-d104c9bbe310\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-hgks7" Jul 09 09:34:34.066166 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:34.066033 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/675a460e-eff5-4cc7-b339-d104c9bbe310-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-hgks7\" (UID: \"675a460e-eff5-4cc7-b339-d104c9bbe310\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-hgks7" Jul 09 09:34:34.069169 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:34.069140 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/675a460e-eff5-4cc7-b339-d104c9bbe310-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-hgks7\" (UID: \"675a460e-eff5-4cc7-b339-d104c9bbe310\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-hgks7" Jul 09 09:34:34.076201 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:34.076175 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmbc4\" (UniqueName: \"kubernetes.io/projected/675a460e-eff5-4cc7-b339-d104c9bbe310-kube-api-access-jmbc4\") pod \"monitoring-plugin-d5dcff7cf-hgks7\" (UID: \"675a460e-eff5-4cc7-b339-d104c9bbe310\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-hgks7" Jul 09 09:34:34.196692 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:34.196621 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-hgks7" Jul 09 09:34:34.360022 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:34.359978 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-hgks7"] Jul 09 09:34:34.364289 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:34:34.364265 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod675a460e_eff5_4cc7_b339_d104c9bbe310.slice/crio-4fa3558db35ef1be4adadd669203ab7d10fda96f66dce587c759d0d09cd25a1a WatchSource:0}: Error finding container 4fa3558db35ef1be4adadd669203ab7d10fda96f66dce587c759d0d09cd25a1a: Status 404 returned error can't find the container with id 4fa3558db35ef1be4adadd669203ab7d10fda96f66dce587c759d0d09cd25a1a Jul 09 09:34:34.801633 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:34.801599 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-88f9f95-bfngx" event={"ID":"c20bfb7c-f525-481e-89c3-2852fce4ffb3","Type":"ContainerStarted","Data":"90824acbb52a83a9196c98120631c4ba854377876c96c709914e578f80443d0f"} Jul 09 09:34:34.803077 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:34.803049 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-hgks7" event={"ID":"675a460e-eff5-4cc7-b339-d104c9bbe310","Type":"ContainerStarted","Data":"4fa3558db35ef1be4adadd669203ab7d10fda96f66dce587c759d0d09cd25a1a"} Jul 09 09:34:35.404527 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.404492 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 09:34:35.415239 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.411286 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:34:35.415792 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.415761 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 09 09:34:35.416053 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.416032 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 09 09:34:35.416263 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.416244 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 09 09:34:35.416475 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.416458 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 09 09:34:35.417035 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.417019 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 09 09:34:35.417362 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.417348 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 09 09:34:35.419415 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.417461 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 09 09:34:35.419415 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.417372 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 09 09:34:35.419415 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.416482 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-2pxlr\"" Jul 09 09:34:35.419415 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.417423 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 09 09:34:35.419415 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.417817 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 09 09:34:35.419415 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.418471 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8r92kmjfma92t\"" Jul 09 09:34:35.419415 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.418665 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 09 09:34:35.419415 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.418911 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 09 09:34:35.419415 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.419133 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 09 09:34:35.421710 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.421681 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 09 09:34:35.451376 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.451025 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 09 09:34:35.464486 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.464469 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 09:34:35.478096 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.478072 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5295084b-def6-431a-b103-b82d9b3b7f74-config-out\") pod \"prometheus-k8s-0\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:34:35.479480 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.479456 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5295084b-def6-431a-b103-b82d9b3b7f74-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:34:35.479661 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.479646 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5295084b-def6-431a-b103-b82d9b3b7f74-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:34:35.479785 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.479768 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5295084b-def6-431a-b103-b82d9b3b7f74-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:34:35.480016 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.479994 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5295084b-def6-431a-b103-b82d9b3b7f74-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:34:35.480148 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.480061 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5295084b-def6-431a-b103-b82d9b3b7f74-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:34:35.480148 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.480089 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5295084b-def6-431a-b103-b82d9b3b7f74-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:34:35.480148 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.480113 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5295084b-def6-431a-b103-b82d9b3b7f74-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:34:35.480148 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.480135 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5295084b-def6-431a-b103-b82d9b3b7f74-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:34:35.480355 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.480167 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5295084b-def6-431a-b103-b82d9b3b7f74-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:34:35.480355 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.480201 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5295084b-def6-431a-b103-b82d9b3b7f74-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:34:35.480355 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.480227 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5295084b-def6-431a-b103-b82d9b3b7f74-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:34:35.480355 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.480254 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5295084b-def6-431a-b103-b82d9b3b7f74-config\") pod \"prometheus-k8s-0\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:34:35.480355 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.480275 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5295084b-def6-431a-b103-b82d9b3b7f74-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:34:35.480355 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.480300 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5295084b-def6-431a-b103-b82d9b3b7f74-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:34:35.480355 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.480343 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5295084b-def6-431a-b103-b82d9b3b7f74-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:34:35.480687 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.480384 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5295084b-def6-431a-b103-b82d9b3b7f74-web-config\") pod \"prometheus-k8s-0\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:34:35.480687 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.480429 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5295084b-def6-431a-b103-b82d9b3b7f74-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:34:35.480687 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.480458 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jqmq\" (UniqueName: \"kubernetes.io/projected/5295084b-def6-431a-b103-b82d9b3b7f74-kube-api-access-8jqmq\") pod \"prometheus-k8s-0\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:34:35.480687 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.480490 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5295084b-def6-431a-b103-b82d9b3b7f74-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:34:35.581962 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.581924 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5295084b-def6-431a-b103-b82d9b3b7f74-web-config\") pod \"prometheus-k8s-0\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:34:35.582130 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.581998 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5295084b-def6-431a-b103-b82d9b3b7f74-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:34:35.582130 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.582031 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jqmq\" (UniqueName: \"kubernetes.io/projected/5295084b-def6-431a-b103-b82d9b3b7f74-kube-api-access-8jqmq\") pod \"prometheus-k8s-0\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:34:35.582130 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.582065 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5295084b-def6-431a-b103-b82d9b3b7f74-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:34:35.582130 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.582101 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5295084b-def6-431a-b103-b82d9b3b7f74-config-out\") pod \"prometheus-k8s-0\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:34:35.582130 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.582122 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5295084b-def6-431a-b103-b82d9b3b7f74-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:34:35.582355 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.582145 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5295084b-def6-431a-b103-b82d9b3b7f74-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:34:35.582355 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.582170 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5295084b-def6-431a-b103-b82d9b3b7f74-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:34:35.582355 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.582194 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5295084b-def6-431a-b103-b82d9b3b7f74-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:34:35.582355 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.582250 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5295084b-def6-431a-b103-b82d9b3b7f74-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:34:35.582355 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.582276 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5295084b-def6-431a-b103-b82d9b3b7f74-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:34:35.582355 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.582299 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5295084b-def6-431a-b103-b82d9b3b7f74-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:34:35.582355 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.582325 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5295084b-def6-431a-b103-b82d9b3b7f74-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:34:35.582355 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.582355 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5295084b-def6-431a-b103-b82d9b3b7f74-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:34:35.582667 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.582387 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5295084b-def6-431a-b103-b82d9b3b7f74-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:34:35.582667 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.582415 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5295084b-def6-431a-b103-b82d9b3b7f74-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:34:35.582667 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.582442 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5295084b-def6-431a-b103-b82d9b3b7f74-config\") pod \"prometheus-k8s-0\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:34:35.582667 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.582466 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5295084b-def6-431a-b103-b82d9b3b7f74-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:34:35.582667 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.582489 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5295084b-def6-431a-b103-b82d9b3b7f74-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:34:35.582667 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.582531 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5295084b-def6-431a-b103-b82d9b3b7f74-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:34:35.583245 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.583175 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5295084b-def6-431a-b103-b82d9b3b7f74-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:34:35.583655 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.583633 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5295084b-def6-431a-b103-b82d9b3b7f74-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:34:35.584207 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.584187 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5295084b-def6-431a-b103-b82d9b3b7f74-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:34:35.585020 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.584645 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5295084b-def6-431a-b103-b82d9b3b7f74-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:34:35.587511 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.587488 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5295084b-def6-431a-b103-b82d9b3b7f74-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:34:35.590534 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.589313 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5295084b-def6-431a-b103-b82d9b3b7f74-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:34:35.590534 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.589467 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5295084b-def6-431a-b103-b82d9b3b7f74-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:34:35.590534 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.589795 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5295084b-def6-431a-b103-b82d9b3b7f74-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:34:35.590534 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.590238 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5295084b-def6-431a-b103-b82d9b3b7f74-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:34:35.592927 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.592900 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5295084b-def6-431a-b103-b82d9b3b7f74-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:34:35.595905 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.593965 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5295084b-def6-431a-b103-b82d9b3b7f74-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:34:35.595905 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.594276 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/5295084b-def6-431a-b103-b82d9b3b7f74-config\") pod \"prometheus-k8s-0\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:34:35.595905 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.594588 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5295084b-def6-431a-b103-b82d9b3b7f74-config-out\") pod \"prometheus-k8s-0\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:34:35.596962 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.596917 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5295084b-def6-431a-b103-b82d9b3b7f74-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:34:35.597373 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.597348 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5295084b-def6-431a-b103-b82d9b3b7f74-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:34:35.597794 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.597775 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5295084b-def6-431a-b103-b82d9b3b7f74-web-config\") pod \"prometheus-k8s-0\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:34:35.597991 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.597971 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5295084b-def6-431a-b103-b82d9b3b7f74-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:34:35.598145 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.598128 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5295084b-def6-431a-b103-b82d9b3b7f74-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:34:35.599292 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.599268 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jqmq\" (UniqueName: \"kubernetes.io/projected/5295084b-def6-431a-b103-b82d9b3b7f74-kube-api-access-8jqmq\") pod \"prometheus-k8s-0\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:34:35.607707 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.607687 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5295084b-def6-431a-b103-b82d9b3b7f74-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:34:35.754497 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:35.754207 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:34:36.437526 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:36.437491 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 09:34:36.694482 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:36.694456 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xf4sw" Jul 09 09:34:36.760080 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:36.760046 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-ztphn" Jul 09 09:34:36.819749 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:36.819710 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-hgks7" event={"ID":"675a460e-eff5-4cc7-b339-d104c9bbe310","Type":"ContainerStarted","Data":"f9e11bea81068273797c6a80c29a1760b299d95cb2f433ecc500aa98ffc11595"} Jul 09 09:34:36.820296 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:36.820276 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-hgks7" Jul 09 09:34:36.825966 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:36.825939 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-hgks7" Jul 09 09:34:36.838592 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:36.838540 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-hgks7" podStartSLOduration=1.894692959 podStartE2EDuration="3.838527866s" totalImagesPullingTime="1.943834907s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:34:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:34:36.83782243 +0000 UTC m=+61.929576350" watchObservedRunningTime="2026-07-09 09:34:36.838527866 +0000 UTC m=+61.930281790" Jul 09 09:34:37.618092 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:34:37.618055 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5295084b_def6_431a_b103_b82d9b3b7f74.slice/crio-05b3e98e2a59b800226671417025bbf59f6c9722a4f2f484b0c7860cd2959037 WatchSource:0}: Error finding container 05b3e98e2a59b800226671417025bbf59f6c9722a4f2f484b0c7860cd2959037: Status 404 returned error can't find the container with id 05b3e98e2a59b800226671417025bbf59f6c9722a4f2f484b0c7860cd2959037 Jul 09 09:34:37.824944 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:37.824899 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-88f9f95-bfngx" event={"ID":"c20bfb7c-f525-481e-89c3-2852fce4ffb3","Type":"ContainerStarted","Data":"80441f272198ec5758cac5c019597b02f104dd8c5236f1c612f60a45c3b895cc"} Jul 09 09:34:37.826301 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:37.826274 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5295084b-def6-431a-b103-b82d9b3b7f74","Type":"ContainerStarted","Data":"05b3e98e2a59b800226671417025bbf59f6c9722a4f2f484b0c7860cd2959037"} Jul 09 09:34:37.848553 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:37.848488 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-88f9f95-bfngx" podStartSLOduration=1.024071296 podStartE2EDuration="4.848470624s" totalImagesPullingTime="3.824399328s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:34:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:34:37.847233901 +0000 UTC m=+62.938987821" watchObservedRunningTime="2026-07-09 09:34:37.848470624 +0000 UTC m=+62.940224546" Jul 09 09:34:38.832561 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:38.832483 2578 generic.go:356] "Generic (PLEG): container finished" podID="5295084b-def6-431a-b103-b82d9b3b7f74" containerID="f55415412e52b7780f0af57911f044b681d0e576326421f10e85c7ae9d6cb9e1" exitCode=0 Jul 09 09:34:38.832973 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:38.832576 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5295084b-def6-431a-b103-b82d9b3b7f74","Type":"ContainerDied","Data":"f55415412e52b7780f0af57911f044b681d0e576326421f10e85c7ae9d6cb9e1"} Jul 09 09:34:40.230769 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:40.230703 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a0c24dc3-5e21-4537-966b-0c5f9e46f8d9-original-pull-secret\") pod \"global-pull-secret-syncer-sw9zq\" (UID: \"a0c24dc3-5e21-4537-966b-0c5f9e46f8d9\") " pod="kube-system/global-pull-secret-syncer-sw9zq" Jul 09 09:34:40.231256 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:40.230798 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0652b01-6ddc-4820-8fbe-c2e5b643f663-metrics-certs\") pod \"network-metrics-daemon-wsdxs\" (UID: \"c0652b01-6ddc-4820-8fbe-c2e5b643f663\") " pod="openshift-multus/network-metrics-daemon-wsdxs" Jul 09 09:34:40.234768 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:40.234734 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 09 09:34:40.234928 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:40.234810 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 09 09:34:40.247423 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:40.247395 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0652b01-6ddc-4820-8fbe-c2e5b643f663-metrics-certs\") pod \"network-metrics-daemon-wsdxs\" (UID: \"c0652b01-6ddc-4820-8fbe-c2e5b643f663\") " pod="openshift-multus/network-metrics-daemon-wsdxs" Jul 09 09:34:40.249206 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:40.249153 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a0c24dc3-5e21-4537-966b-0c5f9e46f8d9-original-pull-secret\") pod \"global-pull-secret-syncer-sw9zq\" (UID: \"a0c24dc3-5e21-4537-966b-0c5f9e46f8d9\") " pod="kube-system/global-pull-secret-syncer-sw9zq" Jul 09 09:34:40.253648 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:40.253407 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sw9zq" Jul 09 09:34:40.331571 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:40.331543 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48bc9\" (UniqueName: \"kubernetes.io/projected/53e2e8a5-34d5-4317-9733-143a53880b9b-kube-api-access-48bc9\") pod \"network-check-target-gqlhc\" (UID: \"53e2e8a5-34d5-4317-9733-143a53880b9b\") " pod="openshift-network-diagnostics/network-check-target-gqlhc" Jul 09 09:34:40.334650 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:40.334595 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 09 09:34:40.344826 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:40.344804 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 09 09:34:40.356891 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:40.356866 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48bc9\" (UniqueName: \"kubernetes.io/projected/53e2e8a5-34d5-4317-9733-143a53880b9b-kube-api-access-48bc9\") pod \"network-check-target-gqlhc\" (UID: \"53e2e8a5-34d5-4317-9733-143a53880b9b\") " pod="openshift-network-diagnostics/network-check-target-gqlhc" Jul 09 09:34:40.451550 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:40.451517 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-btgg6\"" Jul 09 09:34:40.456088 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:40.456063 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sb7qz\"" Jul 09 09:34:40.459279 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:40.459259 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wsdxs" Jul 09 09:34:40.464522 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:40.464500 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gqlhc" Jul 09 09:34:43.639566 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:43.639530 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-88f9f95-bfngx" Jul 09 09:34:43.640037 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:43.639602 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-88f9f95-bfngx" Jul 09 09:34:43.644781 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:43.644759 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-88f9f95-bfngx" Jul 09 09:34:43.856082 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:43.856048 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-88f9f95-bfngx" Jul 09 09:34:49.535456 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:49.535383 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gqlhc"] Jul 09 09:34:49.540046 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:34:49.540002 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53e2e8a5_34d5_4317_9733_143a53880b9b.slice/crio-e1b657351b685cb7c10201e55218a410cca08365dc6cc6a42d1e1e9ba0cdd9f0 WatchSource:0}: Error finding container e1b657351b685cb7c10201e55218a410cca08365dc6cc6a42d1e1e9ba0cdd9f0: Status 404 returned error can't find the container with id e1b657351b685cb7c10201e55218a410cca08365dc6cc6a42d1e1e9ba0cdd9f0 Jul 09 09:34:49.560692 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:49.560666 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wsdxs"] Jul 09 09:34:49.564046 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:34:49.564021 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0652b01_6ddc_4820_8fbe_c2e5b643f663.slice/crio-4c6274d134449857ad10cb435401a5200d5b20ee23d40cacec36f6da4ae371e3 WatchSource:0}: Error finding container 4c6274d134449857ad10cb435401a5200d5b20ee23d40cacec36f6da4ae371e3: Status 404 returned error can't find the container with id 4c6274d134449857ad10cb435401a5200d5b20ee23d40cacec36f6da4ae371e3 Jul 09 09:34:49.577315 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:49.577284 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sw9zq"] Jul 09 09:34:49.580831 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:34:49.580806 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0c24dc3_5e21_4537_966b_0c5f9e46f8d9.slice/crio-db7b90d8364a04ebfba23163136c79e6585988d32f55dd2101c219717f2eeee6 WatchSource:0}: Error finding container db7b90d8364a04ebfba23163136c79e6585988d32f55dd2101c219717f2eeee6: Status 404 returned error can't find the container with id db7b90d8364a04ebfba23163136c79e6585988d32f55dd2101c219717f2eeee6 Jul 09 09:34:49.879236 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:49.879205 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5295084b-def6-431a-b103-b82d9b3b7f74","Type":"ContainerStarted","Data":"5292d0c54d8f0638482519e7964dd6352f53ea14d1bd8582cc6a5dab13ea2899"} Jul 09 09:34:49.879377 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:49.879291 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5295084b-def6-431a-b103-b82d9b3b7f74","Type":"ContainerStarted","Data":"2b480d24de527bffa8b2d866c6bb2ae8be3b7db09c66117f5be3c0b550ad63ce"} Jul 09 09:34:49.880433 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:49.880405 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gqlhc" event={"ID":"53e2e8a5-34d5-4317-9733-143a53880b9b","Type":"ContainerStarted","Data":"e1b657351b685cb7c10201e55218a410cca08365dc6cc6a42d1e1e9ba0cdd9f0"} Jul 09 09:34:49.881560 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:49.881538 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sw9zq" event={"ID":"a0c24dc3-5e21-4537-966b-0c5f9e46f8d9","Type":"ContainerStarted","Data":"db7b90d8364a04ebfba23163136c79e6585988d32f55dd2101c219717f2eeee6"} Jul 09 09:34:49.882673 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:49.882649 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wsdxs" event={"ID":"c0652b01-6ddc-4820-8fbe-c2e5b643f663","Type":"ContainerStarted","Data":"4c6274d134449857ad10cb435401a5200d5b20ee23d40cacec36f6da4ae371e3"} Jul 09 09:34:49.884066 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:49.884045 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-9hk94" event={"ID":"18bcc96c-0950-49b5-b109-fdd52cfd0616","Type":"ContainerStarted","Data":"f3845fca9a9af8515278145c9cec77de26e722e0faec21bfe0d8006c5441a788"} Jul 09 09:34:49.884284 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:49.884261 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-685bfb5f64-9hk94" Jul 09 09:34:49.885522 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:49.885494 2578 patch_prober.go:28] interesting pod/downloads-685bfb5f64-9hk94 container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.133.0.10:8080/\": dial tcp 10.133.0.10:8080: connect: connection refused" start-of-body= Jul 09 09:34:49.885594 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:49.885552 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-685bfb5f64-9hk94" podUID="18bcc96c-0950-49b5-b109-fdd52cfd0616" containerName="download-server" probeResult="failure" output="Get \"http://10.133.0.10:8080/\": dial tcp 10.133.0.10:8080: connect: connection refused" Jul 09 09:34:49.902355 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:49.902286 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-685bfb5f64-9hk94" podStartSLOduration=1.043729862 podStartE2EDuration="21.902273222s" totalImagesPullingTime="20.85854336s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:34:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:34:49.900903117 +0000 UTC m=+74.992657027" watchObservedRunningTime="2026-07-09 09:34:49.902273222 +0000 UTC m=+74.994027142" Jul 09 09:34:50.902680 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:50.902648 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-685bfb5f64-9hk94" Jul 09 09:34:51.919032 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:51.918909 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wsdxs" event={"ID":"c0652b01-6ddc-4820-8fbe-c2e5b643f663","Type":"ContainerStarted","Data":"4141539dbc0a9972693564b41ab56ae389afea01972798eb9fbdd06bd8e5ef27"} Jul 09 09:34:52.929276 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:52.929154 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wsdxs" event={"ID":"c0652b01-6ddc-4820-8fbe-c2e5b643f663","Type":"ContainerStarted","Data":"a7d4c9418bc0a8ba2445bb5a79d1e2ba7e9a6cc016f9124e1c03fccbaf6676e4"} Jul 09 09:34:54.154249 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:54.154190 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-wsdxs" podStartSLOduration=77.09196315 podStartE2EDuration="1m19.154170444s" totalImagesPullingTime="2.062207294s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:33:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:34:52.953162034 +0000 UTC m=+78.044915954" watchObservedRunningTime="2026-07-09 09:34:54.154170444 +0000 UTC m=+79.245924365" Jul 09 09:34:54.156083 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:54.156058 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-88f9f95-bfngx"] Jul 09 09:34:57.954580 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:57.954535 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gqlhc" event={"ID":"53e2e8a5-34d5-4317-9733-143a53880b9b","Type":"ContainerStarted","Data":"ddd7503c4302540a6aa9b52a4a4bc57b7cdbf41b9ac0b5134addce6f9bb01dff"} Jul 09 09:34:57.955144 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:57.954609 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gqlhc" Jul 09 09:34:57.956518 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:57.956487 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sw9zq" event={"ID":"a0c24dc3-5e21-4537-966b-0c5f9e46f8d9","Type":"ContainerStarted","Data":"a63e5e10992bbf7f3196084551e2dafbf0a93b1de79668eb49cf3a171b57d3e4"} Jul 09 09:34:57.960730 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:57.960692 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5295084b-def6-431a-b103-b82d9b3b7f74","Type":"ContainerStarted","Data":"1fbd27e5adc14d3929cbb0cdcc37b1c81a337a8096d510e703d10d76d16c360f"} Jul 09 09:34:57.960877 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:57.960763 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5295084b-def6-431a-b103-b82d9b3b7f74","Type":"ContainerStarted","Data":"79ef3db69361f604e988efea07d21862cf0d46cd5497a3f5e79cab3abab6d5d9"} Jul 09 09:34:57.960877 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:57.960804 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5295084b-def6-431a-b103-b82d9b3b7f74","Type":"ContainerStarted","Data":"89a97192cbcc820f05aac22d089665cc40756048319e25ae36aad2d7c31fd14f"} Jul 09 09:34:57.960877 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:57.960869 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5295084b-def6-431a-b103-b82d9b3b7f74","Type":"ContainerStarted","Data":"6d9df5f195220af70a531f2b0dfafe16f072f8e770a36ad93deddf30da77da1c"} Jul 09 09:34:57.970913 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:57.970864 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-gqlhc" podStartSLOduration=75.428213754 podStartE2EDuration="1m22.970821125s" totalImagesPullingTime="7.542607371s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:33:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:34:57.970168345 +0000 UTC m=+83.061922266" watchObservedRunningTime="2026-07-09 09:34:57.970821125 +0000 UTC m=+83.062575046" Jul 09 09:34:57.984688 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:57.984650 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-sw9zq" podStartSLOduration=75.466197016 podStartE2EDuration="1m22.984636606s" totalImagesPullingTime="7.51843959s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 09:33:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:34:57.983742726 +0000 UTC m=+83.075496647" watchObservedRunningTime="2026-07-09 09:34:57.984636606 +0000 UTC m=+83.076390526" Jul 09 09:34:58.014152 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:34:58.014106 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.091534772 podStartE2EDuration="23.014087734s" totalImagesPullingTime="18.899624432s" totalInitContainerRuntime="22.92853ms" isStatefulPod=false podCreationTimestamp="2026-07-09 09:34:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:34:58.012291214 +0000 UTC m=+83.104045173" watchObservedRunningTime="2026-07-09 09:34:58.014087734 +0000 UTC m=+83.105841655" Jul 09 09:35:00.755296 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:00.755260 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:35:19.182155 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:19.182093 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-88f9f95-bfngx" podUID="c20bfb7c-f525-481e-89c3-2852fce4ffb3" containerName="console" containerID="cri-o://80441f272198ec5758cac5c019597b02f104dd8c5236f1c612f60a45c3b895cc" gracePeriod=15 Jul 09 09:35:19.471545 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:19.471522 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-88f9f95-bfngx" Jul 09 09:35:19.558112 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:19.558087 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c20bfb7c-f525-481e-89c3-2852fce4ffb3-service-ca\") pod \"c20bfb7c-f525-481e-89c3-2852fce4ffb3\" (UID: \"c20bfb7c-f525-481e-89c3-2852fce4ffb3\") " Jul 09 09:35:19.558112 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:19.558116 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c20bfb7c-f525-481e-89c3-2852fce4ffb3-oauth-serving-cert\") pod \"c20bfb7c-f525-481e-89c3-2852fce4ffb3\" (UID: \"c20bfb7c-f525-481e-89c3-2852fce4ffb3\") " Jul 09 09:35:19.558292 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:19.558150 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c20bfb7c-f525-481e-89c3-2852fce4ffb3-console-serving-cert\") pod \"c20bfb7c-f525-481e-89c3-2852fce4ffb3\" (UID: \"c20bfb7c-f525-481e-89c3-2852fce4ffb3\") " Jul 09 09:35:19.558292 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:19.558177 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c20bfb7c-f525-481e-89c3-2852fce4ffb3-console-oauth-config\") pod \"c20bfb7c-f525-481e-89c3-2852fce4ffb3\" (UID: \"c20bfb7c-f525-481e-89c3-2852fce4ffb3\") " Jul 09 09:35:19.558292 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:19.558220 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvhb4\" (UniqueName: \"kubernetes.io/projected/c20bfb7c-f525-481e-89c3-2852fce4ffb3-kube-api-access-gvhb4\") pod \"c20bfb7c-f525-481e-89c3-2852fce4ffb3\" (UID: \"c20bfb7c-f525-481e-89c3-2852fce4ffb3\") " Jul 09 09:35:19.558292 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:19.558257 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c20bfb7c-f525-481e-89c3-2852fce4ffb3-console-config\") pod \"c20bfb7c-f525-481e-89c3-2852fce4ffb3\" (UID: \"c20bfb7c-f525-481e-89c3-2852fce4ffb3\") " Jul 09 09:35:19.558506 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:19.558475 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c20bfb7c-f525-481e-89c3-2852fce4ffb3-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c20bfb7c-f525-481e-89c3-2852fce4ffb3" (UID: "c20bfb7c-f525-481e-89c3-2852fce4ffb3"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 09:35:19.558506 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:19.558495 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c20bfb7c-f525-481e-89c3-2852fce4ffb3-service-ca" (OuterVolumeSpecName: "service-ca") pod "c20bfb7c-f525-481e-89c3-2852fce4ffb3" (UID: "c20bfb7c-f525-481e-89c3-2852fce4ffb3"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 09:35:19.558736 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:19.558717 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c20bfb7c-f525-481e-89c3-2852fce4ffb3-console-config" (OuterVolumeSpecName: "console-config") pod "c20bfb7c-f525-481e-89c3-2852fce4ffb3" (UID: "c20bfb7c-f525-481e-89c3-2852fce4ffb3"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 09:35:19.560619 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:19.560592 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c20bfb7c-f525-481e-89c3-2852fce4ffb3-kube-api-access-gvhb4" (OuterVolumeSpecName: "kube-api-access-gvhb4") pod "c20bfb7c-f525-481e-89c3-2852fce4ffb3" (UID: "c20bfb7c-f525-481e-89c3-2852fce4ffb3"). InnerVolumeSpecName "kube-api-access-gvhb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 09:35:19.560619 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:19.560592 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c20bfb7c-f525-481e-89c3-2852fce4ffb3-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c20bfb7c-f525-481e-89c3-2852fce4ffb3" (UID: "c20bfb7c-f525-481e-89c3-2852fce4ffb3"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 09:35:19.560728 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:19.560653 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c20bfb7c-f525-481e-89c3-2852fce4ffb3-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c20bfb7c-f525-481e-89c3-2852fce4ffb3" (UID: "c20bfb7c-f525-481e-89c3-2852fce4ffb3"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 09:35:19.659482 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:19.659458 2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c20bfb7c-f525-481e-89c3-2852fce4ffb3-service-ca\") on node \"ip-10-0-135-204.ec2.internal\" DevicePath \"\"" Jul 09 09:35:19.659482 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:19.659479 2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c20bfb7c-f525-481e-89c3-2852fce4ffb3-oauth-serving-cert\") on node \"ip-10-0-135-204.ec2.internal\" DevicePath \"\"" Jul 09 09:35:19.659602 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:19.659489 2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c20bfb7c-f525-481e-89c3-2852fce4ffb3-console-serving-cert\") on node \"ip-10-0-135-204.ec2.internal\" DevicePath \"\"" Jul 09 09:35:19.659602 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:19.659499 2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c20bfb7c-f525-481e-89c3-2852fce4ffb3-console-oauth-config\") on node \"ip-10-0-135-204.ec2.internal\" DevicePath \"\"" Jul 09 09:35:19.659602 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:19.659508 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvhb4\" (UniqueName: \"kubernetes.io/projected/c20bfb7c-f525-481e-89c3-2852fce4ffb3-kube-api-access-gvhb4\") on node \"ip-10-0-135-204.ec2.internal\" DevicePath \"\"" Jul 09 09:35:19.659602 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:19.659516 2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c20bfb7c-f525-481e-89c3-2852fce4ffb3-console-config\") on node \"ip-10-0-135-204.ec2.internal\" DevicePath \"\"" Jul 09 09:35:20.040125 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:20.040091 2578 generic.go:356] "Generic (PLEG): container finished" podID="c20bfb7c-f525-481e-89c3-2852fce4ffb3" containerID="80441f272198ec5758cac5c019597b02f104dd8c5236f1c612f60a45c3b895cc" exitCode=0 Jul 09 09:35:20.040268 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:20.040140 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-88f9f95-bfngx" Jul 09 09:35:20.040268 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:20.040177 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-88f9f95-bfngx" event={"ID":"c20bfb7c-f525-481e-89c3-2852fce4ffb3","Type":"ContainerDied","Data":"80441f272198ec5758cac5c019597b02f104dd8c5236f1c612f60a45c3b895cc"} Jul 09 09:35:20.040268 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:20.040227 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-88f9f95-bfngx" event={"ID":"c20bfb7c-f525-481e-89c3-2852fce4ffb3","Type":"ContainerDied","Data":"90824acbb52a83a9196c98120631c4ba854377876c96c709914e578f80443d0f"} Jul 09 09:35:20.040268 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:20.040247 2578 scope.go:117] "RemoveContainer" containerID="80441f272198ec5758cac5c019597b02f104dd8c5236f1c612f60a45c3b895cc" Jul 09 09:35:20.048864 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:20.048831 2578 scope.go:117] "RemoveContainer" containerID="80441f272198ec5758cac5c019597b02f104dd8c5236f1c612f60a45c3b895cc" Jul 09 09:35:20.049119 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:35:20.049100 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80441f272198ec5758cac5c019597b02f104dd8c5236f1c612f60a45c3b895cc\": container with ID starting with 80441f272198ec5758cac5c019597b02f104dd8c5236f1c612f60a45c3b895cc not found: ID does not exist" containerID="80441f272198ec5758cac5c019597b02f104dd8c5236f1c612f60a45c3b895cc" Jul 09 09:35:20.049192 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:20.049129 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80441f272198ec5758cac5c019597b02f104dd8c5236f1c612f60a45c3b895cc"} err="failed to get container status \"80441f272198ec5758cac5c019597b02f104dd8c5236f1c612f60a45c3b895cc\": rpc error: code = NotFound desc = could not find container \"80441f272198ec5758cac5c019597b02f104dd8c5236f1c612f60a45c3b895cc\": container with ID starting with 80441f272198ec5758cac5c019597b02f104dd8c5236f1c612f60a45c3b895cc not found: ID does not exist" Jul 09 09:35:20.060665 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:20.060640 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-88f9f95-bfngx"] Jul 09 09:35:20.066205 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:20.066185 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-88f9f95-bfngx"] Jul 09 09:35:21.442447 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:21.442411 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c20bfb7c-f525-481e-89c3-2852fce4ffb3" path="/var/lib/kubelet/pods/c20bfb7c-f525-481e-89c3-2852fce4ffb3/volumes" Jul 09 09:35:28.966183 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:28.966153 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-gqlhc" Jul 09 09:35:35.755375 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:35.755334 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:35:35.821344 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:35.821318 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:35:36.118306 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:36.118226 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:35:53.683647 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:53.683605 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 09:35:53.684254 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:53.683889 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5295084b-def6-431a-b103-b82d9b3b7f74" containerName="prometheus" containerID="cri-o://2b480d24de527bffa8b2d866c6bb2ae8be3b7db09c66117f5be3c0b550ad63ce" gracePeriod=600 Jul 09 09:35:53.684254 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:53.683931 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5295084b-def6-431a-b103-b82d9b3b7f74" containerName="config-reloader" containerID="cri-o://5292d0c54d8f0638482519e7964dd6352f53ea14d1bd8582cc6a5dab13ea2899" gracePeriod=600 Jul 09 09:35:53.684254 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:53.683980 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5295084b-def6-431a-b103-b82d9b3b7f74" containerName="kube-rbac-proxy-thanos" containerID="cri-o://1fbd27e5adc14d3929cbb0cdcc37b1c81a337a8096d510e703d10d76d16c360f" gracePeriod=600 Jul 09 09:35:53.684254 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:53.683928 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5295084b-def6-431a-b103-b82d9b3b7f74" containerName="kube-rbac-proxy" containerID="cri-o://79ef3db69361f604e988efea07d21862cf0d46cd5497a3f5e79cab3abab6d5d9" gracePeriod=600 Jul 09 09:35:53.684254 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:53.683967 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5295084b-def6-431a-b103-b82d9b3b7f74" containerName="kube-rbac-proxy-web" containerID="cri-o://89a97192cbcc820f05aac22d089665cc40756048319e25ae36aad2d7c31fd14f" gracePeriod=600 Jul 09 09:35:53.684254 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:53.683927 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5295084b-def6-431a-b103-b82d9b3b7f74" containerName="thanos-sidecar" containerID="cri-o://6d9df5f195220af70a531f2b0dfafe16f072f8e770a36ad93deddf30da77da1c" gracePeriod=600 Jul 09 09:35:54.167159 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:54.167130 2578 generic.go:356] "Generic (PLEG): container finished" podID="5295084b-def6-431a-b103-b82d9b3b7f74" containerID="1fbd27e5adc14d3929cbb0cdcc37b1c81a337a8096d510e703d10d76d16c360f" exitCode=0 Jul 09 09:35:54.167159 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:54.167152 2578 generic.go:356] "Generic (PLEG): container finished" podID="5295084b-def6-431a-b103-b82d9b3b7f74" containerID="79ef3db69361f604e988efea07d21862cf0d46cd5497a3f5e79cab3abab6d5d9" exitCode=0 Jul 09 09:35:54.167159 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:54.167160 2578 generic.go:356] "Generic (PLEG): container finished" podID="5295084b-def6-431a-b103-b82d9b3b7f74" containerID="6d9df5f195220af70a531f2b0dfafe16f072f8e770a36ad93deddf30da77da1c" exitCode=0 Jul 09 09:35:54.167159 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:54.167166 2578 generic.go:356] "Generic (PLEG): container finished" podID="5295084b-def6-431a-b103-b82d9b3b7f74" containerID="5292d0c54d8f0638482519e7964dd6352f53ea14d1bd8582cc6a5dab13ea2899" exitCode=0 Jul 09 09:35:54.167406 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:54.167172 2578 generic.go:356] "Generic (PLEG): container finished" podID="5295084b-def6-431a-b103-b82d9b3b7f74" containerID="2b480d24de527bffa8b2d866c6bb2ae8be3b7db09c66117f5be3c0b550ad63ce" exitCode=0 Jul 09 09:35:54.167406 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:54.167206 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5295084b-def6-431a-b103-b82d9b3b7f74","Type":"ContainerDied","Data":"1fbd27e5adc14d3929cbb0cdcc37b1c81a337a8096d510e703d10d76d16c360f"} Jul 09 09:35:54.167406 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:54.167273 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5295084b-def6-431a-b103-b82d9b3b7f74","Type":"ContainerDied","Data":"79ef3db69361f604e988efea07d21862cf0d46cd5497a3f5e79cab3abab6d5d9"} Jul 09 09:35:54.167406 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:54.167300 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5295084b-def6-431a-b103-b82d9b3b7f74","Type":"ContainerDied","Data":"6d9df5f195220af70a531f2b0dfafe16f072f8e770a36ad93deddf30da77da1c"} Jul 09 09:35:54.167406 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:54.167325 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5295084b-def6-431a-b103-b82d9b3b7f74","Type":"ContainerDied","Data":"5292d0c54d8f0638482519e7964dd6352f53ea14d1bd8582cc6a5dab13ea2899"} Jul 09 09:35:54.167406 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:54.167350 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5295084b-def6-431a-b103-b82d9b3b7f74","Type":"ContainerDied","Data":"2b480d24de527bffa8b2d866c6bb2ae8be3b7db09c66117f5be3c0b550ad63ce"} Jul 09 09:35:54.920708 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:54.920688 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:35:55.011203 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.011174 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5295084b-def6-431a-b103-b82d9b3b7f74-secret-kube-rbac-proxy\") pod \"5295084b-def6-431a-b103-b82d9b3b7f74\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " Jul 09 09:35:55.011361 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.011217 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5295084b-def6-431a-b103-b82d9b3b7f74-secret-prometheus-k8s-tls\") pod \"5295084b-def6-431a-b103-b82d9b3b7f74\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " Jul 09 09:35:55.011361 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.011245 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5295084b-def6-431a-b103-b82d9b3b7f74-secret-metrics-client-certs\") pod \"5295084b-def6-431a-b103-b82d9b3b7f74\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " Jul 09 09:35:55.011361 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.011277 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5295084b-def6-431a-b103-b82d9b3b7f74-configmap-metrics-client-ca\") pod \"5295084b-def6-431a-b103-b82d9b3b7f74\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " Jul 09 09:35:55.011361 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.011301 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5295084b-def6-431a-b103-b82d9b3b7f74-prometheus-k8s-rulefiles-2\") pod \"5295084b-def6-431a-b103-b82d9b3b7f74\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " Jul 09 09:35:55.011361 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.011323 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5295084b-def6-431a-b103-b82d9b3b7f74-web-config\") pod \"5295084b-def6-431a-b103-b82d9b3b7f74\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " Jul 09 09:35:55.011361 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.011349 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5295084b-def6-431a-b103-b82d9b3b7f74-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"5295084b-def6-431a-b103-b82d9b3b7f74\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " Jul 09 09:35:55.011665 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.011379 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5295084b-def6-431a-b103-b82d9b3b7f74-prometheus-k8s-rulefiles-0\") pod \"5295084b-def6-431a-b103-b82d9b3b7f74\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " Jul 09 09:35:55.011665 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.011414 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5295084b-def6-431a-b103-b82d9b3b7f74-tls-assets\") pod \"5295084b-def6-431a-b103-b82d9b3b7f74\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " Jul 09 09:35:55.011665 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.011440 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5295084b-def6-431a-b103-b82d9b3b7f74-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"5295084b-def6-431a-b103-b82d9b3b7f74\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " Jul 09 09:35:55.011665 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.011470 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5295084b-def6-431a-b103-b82d9b3b7f74-configmap-kubelet-serving-ca-bundle\") pod \"5295084b-def6-431a-b103-b82d9b3b7f74\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " Jul 09 09:35:55.011665 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.011532 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5295084b-def6-431a-b103-b82d9b3b7f74-configmap-serving-certs-ca-bundle\") pod \"5295084b-def6-431a-b103-b82d9b3b7f74\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " Jul 09 09:35:55.011665 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.011573 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5295084b-def6-431a-b103-b82d9b3b7f74-config-out\") pod \"5295084b-def6-431a-b103-b82d9b3b7f74\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " Jul 09 09:35:55.011665 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.011621 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5295084b-def6-431a-b103-b82d9b3b7f74-config\") pod \"5295084b-def6-431a-b103-b82d9b3b7f74\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " Jul 09 09:35:55.011665 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.011649 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8jqmq\" (UniqueName: \"kubernetes.io/projected/5295084b-def6-431a-b103-b82d9b3b7f74-kube-api-access-8jqmq\") pod \"5295084b-def6-431a-b103-b82d9b3b7f74\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " Jul 09 09:35:55.011665 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.011650 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5295084b-def6-431a-b103-b82d9b3b7f74-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "5295084b-def6-431a-b103-b82d9b3b7f74" (UID: "5295084b-def6-431a-b103-b82d9b3b7f74"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 09:35:55.011665 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.011641 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5295084b-def6-431a-b103-b82d9b3b7f74-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "5295084b-def6-431a-b103-b82d9b3b7f74" (UID: "5295084b-def6-431a-b103-b82d9b3b7f74"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 09:35:55.012142 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.011688 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5295084b-def6-431a-b103-b82d9b3b7f74-secret-grpc-tls\") pod \"5295084b-def6-431a-b103-b82d9b3b7f74\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " Jul 09 09:35:55.012142 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.011717 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5295084b-def6-431a-b103-b82d9b3b7f74-prometheus-k8s-rulefiles-1\") pod \"5295084b-def6-431a-b103-b82d9b3b7f74\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " Jul 09 09:35:55.012142 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.011751 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5295084b-def6-431a-b103-b82d9b3b7f74-thanos-prometheus-http-client-file\") pod \"5295084b-def6-431a-b103-b82d9b3b7f74\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " Jul 09 09:35:55.012142 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.011777 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5295084b-def6-431a-b103-b82d9b3b7f74-prometheus-trusted-ca-bundle\") pod \"5295084b-def6-431a-b103-b82d9b3b7f74\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " Jul 09 09:35:55.012142 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.011818 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5295084b-def6-431a-b103-b82d9b3b7f74-prometheus-k8s-db\") pod \"5295084b-def6-431a-b103-b82d9b3b7f74\" (UID: \"5295084b-def6-431a-b103-b82d9b3b7f74\") " Jul 09 09:35:55.012142 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.012049 2578 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5295084b-def6-431a-b103-b82d9b3b7f74-configmap-metrics-client-ca\") on node \"ip-10-0-135-204.ec2.internal\" DevicePath \"\"" Jul 09 09:35:55.012142 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.012067 2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5295084b-def6-431a-b103-b82d9b3b7f74-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-135-204.ec2.internal\" DevicePath \"\"" Jul 09 09:35:55.013044 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.012957 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5295084b-def6-431a-b103-b82d9b3b7f74-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "5295084b-def6-431a-b103-b82d9b3b7f74" (UID: "5295084b-def6-431a-b103-b82d9b3b7f74"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 09:35:55.014507 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.014267 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5295084b-def6-431a-b103-b82d9b3b7f74-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "5295084b-def6-431a-b103-b82d9b3b7f74" (UID: "5295084b-def6-431a-b103-b82d9b3b7f74"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 09:35:55.014507 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.014306 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5295084b-def6-431a-b103-b82d9b3b7f74-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "5295084b-def6-431a-b103-b82d9b3b7f74" (UID: "5295084b-def6-431a-b103-b82d9b3b7f74"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 09:35:55.014507 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.014491 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5295084b-def6-431a-b103-b82d9b3b7f74-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "5295084b-def6-431a-b103-b82d9b3b7f74" (UID: "5295084b-def6-431a-b103-b82d9b3b7f74"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 09:35:55.014730 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.014578 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5295084b-def6-431a-b103-b82d9b3b7f74-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "5295084b-def6-431a-b103-b82d9b3b7f74" (UID: "5295084b-def6-431a-b103-b82d9b3b7f74"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 09:35:55.014730 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.014612 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5295084b-def6-431a-b103-b82d9b3b7f74-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "5295084b-def6-431a-b103-b82d9b3b7f74" (UID: "5295084b-def6-431a-b103-b82d9b3b7f74"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 09:35:55.014874 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.014747 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5295084b-def6-431a-b103-b82d9b3b7f74-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "5295084b-def6-431a-b103-b82d9b3b7f74" (UID: "5295084b-def6-431a-b103-b82d9b3b7f74"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 09:35:55.015901 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.015272 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5295084b-def6-431a-b103-b82d9b3b7f74-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "5295084b-def6-431a-b103-b82d9b3b7f74" (UID: "5295084b-def6-431a-b103-b82d9b3b7f74"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 09:35:55.015901 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.015596 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5295084b-def6-431a-b103-b82d9b3b7f74-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "5295084b-def6-431a-b103-b82d9b3b7f74" (UID: "5295084b-def6-431a-b103-b82d9b3b7f74"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 09:35:55.016062 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.015919 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5295084b-def6-431a-b103-b82d9b3b7f74-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "5295084b-def6-431a-b103-b82d9b3b7f74" (UID: "5295084b-def6-431a-b103-b82d9b3b7f74"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 09:35:55.016062 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.015936 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5295084b-def6-431a-b103-b82d9b3b7f74-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "5295084b-def6-431a-b103-b82d9b3b7f74" (UID: "5295084b-def6-431a-b103-b82d9b3b7f74"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 09:35:55.016283 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.016245 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5295084b-def6-431a-b103-b82d9b3b7f74-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "5295084b-def6-431a-b103-b82d9b3b7f74" (UID: "5295084b-def6-431a-b103-b82d9b3b7f74"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 09:35:55.016608 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.016578 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5295084b-def6-431a-b103-b82d9b3b7f74-config-out" (OuterVolumeSpecName: "config-out") pod "5295084b-def6-431a-b103-b82d9b3b7f74" (UID: "5295084b-def6-431a-b103-b82d9b3b7f74"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 09:35:55.017067 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.017043 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5295084b-def6-431a-b103-b82d9b3b7f74-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "5295084b-def6-431a-b103-b82d9b3b7f74" (UID: "5295084b-def6-431a-b103-b82d9b3b7f74"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 09:35:55.017173 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.017096 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5295084b-def6-431a-b103-b82d9b3b7f74-config" (OuterVolumeSpecName: "config") pod "5295084b-def6-431a-b103-b82d9b3b7f74" (UID: "5295084b-def6-431a-b103-b82d9b3b7f74"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 09:35:55.017340 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.017323 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5295084b-def6-431a-b103-b82d9b3b7f74-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "5295084b-def6-431a-b103-b82d9b3b7f74" (UID: "5295084b-def6-431a-b103-b82d9b3b7f74"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 09:35:55.017877 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.017861 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5295084b-def6-431a-b103-b82d9b3b7f74-kube-api-access-8jqmq" (OuterVolumeSpecName: "kube-api-access-8jqmq") pod "5295084b-def6-431a-b103-b82d9b3b7f74" (UID: "5295084b-def6-431a-b103-b82d9b3b7f74"). InnerVolumeSpecName "kube-api-access-8jqmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 09:35:55.026224 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.026197 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5295084b-def6-431a-b103-b82d9b3b7f74-web-config" (OuterVolumeSpecName: "web-config") pod "5295084b-def6-431a-b103-b82d9b3b7f74" (UID: "5295084b-def6-431a-b103-b82d9b3b7f74"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 09:35:55.113153 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.113083 2578 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5295084b-def6-431a-b103-b82d9b3b7f74-secret-grpc-tls\") on node \"ip-10-0-135-204.ec2.internal\" DevicePath \"\"" Jul 09 09:35:55.113153 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.113109 2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5295084b-def6-431a-b103-b82d9b3b7f74-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-135-204.ec2.internal\" DevicePath \"\"" Jul 09 09:35:55.113153 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.113121 2578 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5295084b-def6-431a-b103-b82d9b3b7f74-thanos-prometheus-http-client-file\") on node \"ip-10-0-135-204.ec2.internal\" DevicePath \"\"" Jul 09 09:35:55.113153 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.113130 2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5295084b-def6-431a-b103-b82d9b3b7f74-prometheus-trusted-ca-bundle\") on node \"ip-10-0-135-204.ec2.internal\" DevicePath \"\"" Jul 09 09:35:55.113153 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.113138 2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5295084b-def6-431a-b103-b82d9b3b7f74-prometheus-k8s-db\") on node \"ip-10-0-135-204.ec2.internal\" DevicePath \"\"" Jul 09 09:35:55.113153 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.113147 2578 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5295084b-def6-431a-b103-b82d9b3b7f74-secret-kube-rbac-proxy\") on node \"ip-10-0-135-204.ec2.internal\" DevicePath \"\"" Jul 09 09:35:55.113153 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.113155 2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5295084b-def6-431a-b103-b82d9b3b7f74-secret-prometheus-k8s-tls\") on node \"ip-10-0-135-204.ec2.internal\" DevicePath \"\"" Jul 09 09:35:55.113436 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.113163 2578 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5295084b-def6-431a-b103-b82d9b3b7f74-secret-metrics-client-certs\") on node \"ip-10-0-135-204.ec2.internal\" DevicePath \"\"" Jul 09 09:35:55.113436 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.113172 2578 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5295084b-def6-431a-b103-b82d9b3b7f74-web-config\") on node \"ip-10-0-135-204.ec2.internal\" DevicePath \"\"" Jul 09 09:35:55.113436 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.113181 2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5295084b-def6-431a-b103-b82d9b3b7f74-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-135-204.ec2.internal\" DevicePath \"\"" Jul 09 09:35:55.113436 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.113189 2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5295084b-def6-431a-b103-b82d9b3b7f74-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-135-204.ec2.internal\" DevicePath \"\"" Jul 09 09:35:55.113436 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.113197 2578 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5295084b-def6-431a-b103-b82d9b3b7f74-tls-assets\") on node \"ip-10-0-135-204.ec2.internal\" DevicePath \"\"" Jul 09 09:35:55.113436 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.113206 2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5295084b-def6-431a-b103-b82d9b3b7f74-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-135-204.ec2.internal\" DevicePath \"\"" Jul 09 09:35:55.113436 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.113214 2578 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5295084b-def6-431a-b103-b82d9b3b7f74-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-135-204.ec2.internal\" DevicePath \"\"" Jul 09 09:35:55.113436 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.113222 2578 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5295084b-def6-431a-b103-b82d9b3b7f74-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-135-204.ec2.internal\" DevicePath \"\"" Jul 09 09:35:55.113436 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.113230 2578 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5295084b-def6-431a-b103-b82d9b3b7f74-config-out\") on node \"ip-10-0-135-204.ec2.internal\" DevicePath \"\"" Jul 09 09:35:55.113436 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.113237 2578 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/5295084b-def6-431a-b103-b82d9b3b7f74-config\") on node \"ip-10-0-135-204.ec2.internal\" DevicePath \"\"" Jul 09 09:35:55.113436 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.113245 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jqmq\" (UniqueName: \"kubernetes.io/projected/5295084b-def6-431a-b103-b82d9b3b7f74-kube-api-access-8jqmq\") on node \"ip-10-0-135-204.ec2.internal\" DevicePath \"\"" Jul 09 09:35:55.173351 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.173314 2578 generic.go:356] "Generic (PLEG): container finished" podID="5295084b-def6-431a-b103-b82d9b3b7f74" containerID="89a97192cbcc820f05aac22d089665cc40756048319e25ae36aad2d7c31fd14f" exitCode=0 Jul 09 09:35:55.173447 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.173359 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5295084b-def6-431a-b103-b82d9b3b7f74","Type":"ContainerDied","Data":"89a97192cbcc820f05aac22d089665cc40756048319e25ae36aad2d7c31fd14f"} Jul 09 09:35:55.173447 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.173404 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:35:55.173447 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.173431 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5295084b-def6-431a-b103-b82d9b3b7f74","Type":"ContainerDied","Data":"05b3e98e2a59b800226671417025bbf59f6c9722a4f2f484b0c7860cd2959037"} Jul 09 09:35:55.173553 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.173483 2578 scope.go:117] "RemoveContainer" containerID="1fbd27e5adc14d3929cbb0cdcc37b1c81a337a8096d510e703d10d76d16c360f" Jul 09 09:35:55.181376 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.181248 2578 scope.go:117] "RemoveContainer" containerID="79ef3db69361f604e988efea07d21862cf0d46cd5497a3f5e79cab3abab6d5d9" Jul 09 09:35:55.187801 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.187774 2578 scope.go:117] "RemoveContainer" containerID="89a97192cbcc820f05aac22d089665cc40756048319e25ae36aad2d7c31fd14f" Jul 09 09:35:55.194073 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.194054 2578 scope.go:117] "RemoveContainer" containerID="6d9df5f195220af70a531f2b0dfafe16f072f8e770a36ad93deddf30da77da1c" Jul 09 09:35:55.197078 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.197057 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 09:35:55.200827 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.200808 2578 scope.go:117] "RemoveContainer" containerID="5292d0c54d8f0638482519e7964dd6352f53ea14d1bd8582cc6a5dab13ea2899" Jul 09 09:35:55.202330 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.202316 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 09:35:55.207539 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.207522 2578 scope.go:117] "RemoveContainer" containerID="2b480d24de527bffa8b2d866c6bb2ae8be3b7db09c66117f5be3c0b550ad63ce" Jul 09 09:35:55.218090 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.218073 2578 scope.go:117] "RemoveContainer" containerID="f55415412e52b7780f0af57911f044b681d0e576326421f10e85c7ae9d6cb9e1" Jul 09 09:35:55.224417 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.224401 2578 scope.go:117] "RemoveContainer" containerID="1fbd27e5adc14d3929cbb0cdcc37b1c81a337a8096d510e703d10d76d16c360f" Jul 09 09:35:55.224678 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:35:55.224660 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fbd27e5adc14d3929cbb0cdcc37b1c81a337a8096d510e703d10d76d16c360f\": container with ID starting with 1fbd27e5adc14d3929cbb0cdcc37b1c81a337a8096d510e703d10d76d16c360f not found: ID does not exist" containerID="1fbd27e5adc14d3929cbb0cdcc37b1c81a337a8096d510e703d10d76d16c360f" Jul 09 09:35:55.224753 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.224689 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fbd27e5adc14d3929cbb0cdcc37b1c81a337a8096d510e703d10d76d16c360f"} err="failed to get container status \"1fbd27e5adc14d3929cbb0cdcc37b1c81a337a8096d510e703d10d76d16c360f\": rpc error: code = NotFound desc = could not find container \"1fbd27e5adc14d3929cbb0cdcc37b1c81a337a8096d510e703d10d76d16c360f\": container with ID starting with 1fbd27e5adc14d3929cbb0cdcc37b1c81a337a8096d510e703d10d76d16c360f not found: ID does not exist" Jul 09 09:35:55.224753 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.224713 2578 scope.go:117] "RemoveContainer" containerID="79ef3db69361f604e988efea07d21862cf0d46cd5497a3f5e79cab3abab6d5d9" Jul 09 09:35:55.224980 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:35:55.224962 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79ef3db69361f604e988efea07d21862cf0d46cd5497a3f5e79cab3abab6d5d9\": container with ID starting with 79ef3db69361f604e988efea07d21862cf0d46cd5497a3f5e79cab3abab6d5d9 not found: ID does not exist" containerID="79ef3db69361f604e988efea07d21862cf0d46cd5497a3f5e79cab3abab6d5d9" Jul 09 09:35:55.225056 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.224990 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79ef3db69361f604e988efea07d21862cf0d46cd5497a3f5e79cab3abab6d5d9"} err="failed to get container status \"79ef3db69361f604e988efea07d21862cf0d46cd5497a3f5e79cab3abab6d5d9\": rpc error: code = NotFound desc = could not find container \"79ef3db69361f604e988efea07d21862cf0d46cd5497a3f5e79cab3abab6d5d9\": container with ID starting with 79ef3db69361f604e988efea07d21862cf0d46cd5497a3f5e79cab3abab6d5d9 not found: ID does not exist" Jul 09 09:35:55.225056 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.225010 2578 scope.go:117] "RemoveContainer" containerID="89a97192cbcc820f05aac22d089665cc40756048319e25ae36aad2d7c31fd14f" Jul 09 09:35:55.225229 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:35:55.225209 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89a97192cbcc820f05aac22d089665cc40756048319e25ae36aad2d7c31fd14f\": container with ID starting with 89a97192cbcc820f05aac22d089665cc40756048319e25ae36aad2d7c31fd14f not found: ID does not exist" containerID="89a97192cbcc820f05aac22d089665cc40756048319e25ae36aad2d7c31fd14f" Jul 09 09:35:55.225273 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.225235 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89a97192cbcc820f05aac22d089665cc40756048319e25ae36aad2d7c31fd14f"} err="failed to get container status \"89a97192cbcc820f05aac22d089665cc40756048319e25ae36aad2d7c31fd14f\": rpc error: code = NotFound desc = could not find container \"89a97192cbcc820f05aac22d089665cc40756048319e25ae36aad2d7c31fd14f\": container with ID starting with 89a97192cbcc820f05aac22d089665cc40756048319e25ae36aad2d7c31fd14f not found: ID does not exist" Jul 09 09:35:55.225273 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.225250 2578 scope.go:117] "RemoveContainer" containerID="6d9df5f195220af70a531f2b0dfafe16f072f8e770a36ad93deddf30da77da1c" Jul 09 09:35:55.225455 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:35:55.225439 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d9df5f195220af70a531f2b0dfafe16f072f8e770a36ad93deddf30da77da1c\": container with ID starting with 6d9df5f195220af70a531f2b0dfafe16f072f8e770a36ad93deddf30da77da1c not found: ID does not exist" containerID="6d9df5f195220af70a531f2b0dfafe16f072f8e770a36ad93deddf30da77da1c" Jul 09 09:35:55.225515 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.225462 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d9df5f195220af70a531f2b0dfafe16f072f8e770a36ad93deddf30da77da1c"} err="failed to get container status \"6d9df5f195220af70a531f2b0dfafe16f072f8e770a36ad93deddf30da77da1c\": rpc error: code = NotFound desc = could not find container \"6d9df5f195220af70a531f2b0dfafe16f072f8e770a36ad93deddf30da77da1c\": container with ID starting with 6d9df5f195220af70a531f2b0dfafe16f072f8e770a36ad93deddf30da77da1c not found: ID does not exist" Jul 09 09:35:55.225515 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.225479 2578 scope.go:117] "RemoveContainer" containerID="5292d0c54d8f0638482519e7964dd6352f53ea14d1bd8582cc6a5dab13ea2899" Jul 09 09:35:55.225666 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:35:55.225652 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5292d0c54d8f0638482519e7964dd6352f53ea14d1bd8582cc6a5dab13ea2899\": container with ID starting with 5292d0c54d8f0638482519e7964dd6352f53ea14d1bd8582cc6a5dab13ea2899 not found: ID does not exist" containerID="5292d0c54d8f0638482519e7964dd6352f53ea14d1bd8582cc6a5dab13ea2899" Jul 09 09:35:55.225704 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.225669 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5292d0c54d8f0638482519e7964dd6352f53ea14d1bd8582cc6a5dab13ea2899"} err="failed to get container status \"5292d0c54d8f0638482519e7964dd6352f53ea14d1bd8582cc6a5dab13ea2899\": rpc error: code = NotFound desc = could not find container \"5292d0c54d8f0638482519e7964dd6352f53ea14d1bd8582cc6a5dab13ea2899\": container with ID starting with 5292d0c54d8f0638482519e7964dd6352f53ea14d1bd8582cc6a5dab13ea2899 not found: ID does not exist" Jul 09 09:35:55.225704 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.225681 2578 scope.go:117] "RemoveContainer" containerID="2b480d24de527bffa8b2d866c6bb2ae8be3b7db09c66117f5be3c0b550ad63ce" Jul 09 09:35:55.225882 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:35:55.225864 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b480d24de527bffa8b2d866c6bb2ae8be3b7db09c66117f5be3c0b550ad63ce\": container with ID starting with 2b480d24de527bffa8b2d866c6bb2ae8be3b7db09c66117f5be3c0b550ad63ce not found: ID does not exist" containerID="2b480d24de527bffa8b2d866c6bb2ae8be3b7db09c66117f5be3c0b550ad63ce" Jul 09 09:35:55.225926 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.225887 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b480d24de527bffa8b2d866c6bb2ae8be3b7db09c66117f5be3c0b550ad63ce"} err="failed to get container status \"2b480d24de527bffa8b2d866c6bb2ae8be3b7db09c66117f5be3c0b550ad63ce\": rpc error: code = NotFound desc = could not find container \"2b480d24de527bffa8b2d866c6bb2ae8be3b7db09c66117f5be3c0b550ad63ce\": container with ID starting with 2b480d24de527bffa8b2d866c6bb2ae8be3b7db09c66117f5be3c0b550ad63ce not found: ID does not exist" Jul 09 09:35:55.225926 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.225901 2578 scope.go:117] "RemoveContainer" containerID="f55415412e52b7780f0af57911f044b681d0e576326421f10e85c7ae9d6cb9e1" Jul 09 09:35:55.226106 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:35:55.226089 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f55415412e52b7780f0af57911f044b681d0e576326421f10e85c7ae9d6cb9e1\": container with ID starting with f55415412e52b7780f0af57911f044b681d0e576326421f10e85c7ae9d6cb9e1 not found: ID does not exist" containerID="f55415412e52b7780f0af57911f044b681d0e576326421f10e85c7ae9d6cb9e1" Jul 09 09:35:55.226146 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.226111 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f55415412e52b7780f0af57911f044b681d0e576326421f10e85c7ae9d6cb9e1"} err="failed to get container status \"f55415412e52b7780f0af57911f044b681d0e576326421f10e85c7ae9d6cb9e1\": rpc error: code = NotFound desc = could not find container \"f55415412e52b7780f0af57911f044b681d0e576326421f10e85c7ae9d6cb9e1\": container with ID starting with f55415412e52b7780f0af57911f044b681d0e576326421f10e85c7ae9d6cb9e1 not found: ID does not exist" Jul 09 09:35:55.234421 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.234402 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 09:35:55.234636 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.234624 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5295084b-def6-431a-b103-b82d9b3b7f74" containerName="kube-rbac-proxy-thanos" Jul 09 09:35:55.234689 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.234638 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5295084b-def6-431a-b103-b82d9b3b7f74" containerName="kube-rbac-proxy-thanos" Jul 09 09:35:55.234689 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.234647 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5295084b-def6-431a-b103-b82d9b3b7f74" containerName="prometheus" Jul 09 09:35:55.234689 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.234653 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5295084b-def6-431a-b103-b82d9b3b7f74" containerName="prometheus" Jul 09 09:35:55.234689 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.234663 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5295084b-def6-431a-b103-b82d9b3b7f74" containerName="thanos-sidecar" Jul 09 09:35:55.234689 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.234668 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5295084b-def6-431a-b103-b82d9b3b7f74" containerName="thanos-sidecar" Jul 09 09:35:55.234689 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.234675 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5295084b-def6-431a-b103-b82d9b3b7f74" containerName="init-config-reloader" Jul 09 09:35:55.234689 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.234680 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5295084b-def6-431a-b103-b82d9b3b7f74" containerName="init-config-reloader" Jul 09 09:35:55.234689 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.234685 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5295084b-def6-431a-b103-b82d9b3b7f74" containerName="config-reloader" Jul 09 09:35:55.234689 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.234690 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5295084b-def6-431a-b103-b82d9b3b7f74" containerName="config-reloader" Jul 09 09:35:55.235007 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.234698 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c20bfb7c-f525-481e-89c3-2852fce4ffb3" containerName="console" Jul 09 09:35:55.235007 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.234703 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c20bfb7c-f525-481e-89c3-2852fce4ffb3" containerName="console" Jul 09 09:35:55.235007 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.234709 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5295084b-def6-431a-b103-b82d9b3b7f74" containerName="kube-rbac-proxy-web" Jul 09 09:35:55.235007 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.234714 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5295084b-def6-431a-b103-b82d9b3b7f74" containerName="kube-rbac-proxy-web" Jul 09 09:35:55.235007 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.234720 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5295084b-def6-431a-b103-b82d9b3b7f74" containerName="kube-rbac-proxy" Jul 09 09:35:55.235007 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.234724 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5295084b-def6-431a-b103-b82d9b3b7f74" containerName="kube-rbac-proxy" Jul 09 09:35:55.235007 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.234762 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5295084b-def6-431a-b103-b82d9b3b7f74" containerName="thanos-sidecar" Jul 09 09:35:55.235007 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.234769 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5295084b-def6-431a-b103-b82d9b3b7f74" containerName="kube-rbac-proxy-thanos" Jul 09 09:35:55.235007 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.234776 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5295084b-def6-431a-b103-b82d9b3b7f74" containerName="prometheus" Jul 09 09:35:55.235007 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.234781 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5295084b-def6-431a-b103-b82d9b3b7f74" containerName="config-reloader" Jul 09 09:35:55.235007 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.234785 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5295084b-def6-431a-b103-b82d9b3b7f74" containerName="kube-rbac-proxy" Jul 09 09:35:55.235007 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.234790 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c20bfb7c-f525-481e-89c3-2852fce4ffb3" containerName="console" Jul 09 09:35:55.235007 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.234794 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5295084b-def6-431a-b103-b82d9b3b7f74" containerName="kube-rbac-proxy-web" Jul 09 09:35:55.239809 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.239781 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:35:55.242409 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.242389 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 09 09:35:55.242493 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.242427 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8r92kmjfma92t\"" Jul 09 09:35:55.242719 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.242703 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 09 09:35:55.242803 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.242741 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 09 09:35:55.242803 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.242790 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 09 09:35:55.242951 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.242800 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 09 09:35:55.242951 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.242857 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 09 09:35:55.242951 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.242883 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 09 09:35:55.243227 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.243203 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 09 09:35:55.243305 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.243202 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-2pxlr\"" Jul 09 09:35:55.243305 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.243249 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 09 09:35:55.243418 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.243346 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 09 09:35:55.243897 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.243879 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 09 09:35:55.244572 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.244556 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 09 09:35:55.244879 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.244862 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 09 09:35:55.245861 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.245827 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 09 09:35:55.249607 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.249591 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 09 09:35:55.254314 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.254295 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 09:35:55.314339 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.314314 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cb032b45-8976-4ea6-9b05-1dbb6e6ba94d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cb032b45-8976-4ea6-9b05-1dbb6e6ba94d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:35:55.314450 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.314342 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cb032b45-8976-4ea6-9b05-1dbb6e6ba94d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cb032b45-8976-4ea6-9b05-1dbb6e6ba94d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:35:55.314450 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.314366 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cb032b45-8976-4ea6-9b05-1dbb6e6ba94d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cb032b45-8976-4ea6-9b05-1dbb6e6ba94d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:35:55.314450 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.314381 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cb032b45-8976-4ea6-9b05-1dbb6e6ba94d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cb032b45-8976-4ea6-9b05-1dbb6e6ba94d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:35:55.314450 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.314404 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb032b45-8976-4ea6-9b05-1dbb6e6ba94d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cb032b45-8976-4ea6-9b05-1dbb6e6ba94d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:35:55.314450 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.314443 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cb032b45-8976-4ea6-9b05-1dbb6e6ba94d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cb032b45-8976-4ea6-9b05-1dbb6e6ba94d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:35:55.314638 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.314469 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cb032b45-8976-4ea6-9b05-1dbb6e6ba94d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cb032b45-8976-4ea6-9b05-1dbb6e6ba94d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:35:55.314638 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.314486 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cb032b45-8976-4ea6-9b05-1dbb6e6ba94d-web-config\") pod \"prometheus-k8s-0\" (UID: \"cb032b45-8976-4ea6-9b05-1dbb6e6ba94d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:35:55.314638 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.314504 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cb032b45-8976-4ea6-9b05-1dbb6e6ba94d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cb032b45-8976-4ea6-9b05-1dbb6e6ba94d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:35:55.314638 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.314519 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/cb032b45-8976-4ea6-9b05-1dbb6e6ba94d-config\") pod \"prometheus-k8s-0\" (UID: \"cb032b45-8976-4ea6-9b05-1dbb6e6ba94d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:35:55.314638 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.314539 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cb032b45-8976-4ea6-9b05-1dbb6e6ba94d-config-out\") pod \"prometheus-k8s-0\" (UID: \"cb032b45-8976-4ea6-9b05-1dbb6e6ba94d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:35:55.314638 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.314603 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cb032b45-8976-4ea6-9b05-1dbb6e6ba94d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cb032b45-8976-4ea6-9b05-1dbb6e6ba94d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:35:55.314899 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.314651 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb032b45-8976-4ea6-9b05-1dbb6e6ba94d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cb032b45-8976-4ea6-9b05-1dbb6e6ba94d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:35:55.314899 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.314671 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cb032b45-8976-4ea6-9b05-1dbb6e6ba94d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cb032b45-8976-4ea6-9b05-1dbb6e6ba94d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:35:55.314899 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.314686 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cb032b45-8976-4ea6-9b05-1dbb6e6ba94d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cb032b45-8976-4ea6-9b05-1dbb6e6ba94d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:35:55.314899 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.314703 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb032b45-8976-4ea6-9b05-1dbb6e6ba94d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cb032b45-8976-4ea6-9b05-1dbb6e6ba94d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:35:55.314899 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.314725 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cb032b45-8976-4ea6-9b05-1dbb6e6ba94d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cb032b45-8976-4ea6-9b05-1dbb6e6ba94d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:35:55.314899 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.314757 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cb032b45-8976-4ea6-9b05-1dbb6e6ba94d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cb032b45-8976-4ea6-9b05-1dbb6e6ba94d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:35:55.314899 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.314797 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cb032b45-8976-4ea6-9b05-1dbb6e6ba94d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cb032b45-8976-4ea6-9b05-1dbb6e6ba94d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:35:55.314899 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.314863 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwmsb\" (UniqueName: \"kubernetes.io/projected/cb032b45-8976-4ea6-9b05-1dbb6e6ba94d-kube-api-access-xwmsb\") pod \"prometheus-k8s-0\" (UID: \"cb032b45-8976-4ea6-9b05-1dbb6e6ba94d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:35:55.416171 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.416134 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cb032b45-8976-4ea6-9b05-1dbb6e6ba94d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cb032b45-8976-4ea6-9b05-1dbb6e6ba94d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:35:55.416295 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.416179 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cb032b45-8976-4ea6-9b05-1dbb6e6ba94d-web-config\") pod \"prometheus-k8s-0\" (UID: \"cb032b45-8976-4ea6-9b05-1dbb6e6ba94d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:35:55.416295 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.416205 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cb032b45-8976-4ea6-9b05-1dbb6e6ba94d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cb032b45-8976-4ea6-9b05-1dbb6e6ba94d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:35:55.416295 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.416228 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/cb032b45-8976-4ea6-9b05-1dbb6e6ba94d-config\") pod \"prometheus-k8s-0\" (UID: \"cb032b45-8976-4ea6-9b05-1dbb6e6ba94d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:35:55.416295 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.416255 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cb032b45-8976-4ea6-9b05-1dbb6e6ba94d-config-out\") pod \"prometheus-k8s-0\" (UID: \"cb032b45-8976-4ea6-9b05-1dbb6e6ba94d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:35:55.416295 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.416281 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cb032b45-8976-4ea6-9b05-1dbb6e6ba94d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cb032b45-8976-4ea6-9b05-1dbb6e6ba94d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:35:55.416529 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.416329 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb032b45-8976-4ea6-9b05-1dbb6e6ba94d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cb032b45-8976-4ea6-9b05-1dbb6e6ba94d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:35:55.416529 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.416379 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cb032b45-8976-4ea6-9b05-1dbb6e6ba94d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cb032b45-8976-4ea6-9b05-1dbb6e6ba94d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:35:55.416529 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.416409 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cb032b45-8976-4ea6-9b05-1dbb6e6ba94d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cb032b45-8976-4ea6-9b05-1dbb6e6ba94d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:35:55.416529 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.416436 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb032b45-8976-4ea6-9b05-1dbb6e6ba94d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cb032b45-8976-4ea6-9b05-1dbb6e6ba94d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:35:55.416529 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.416471 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cb032b45-8976-4ea6-9b05-1dbb6e6ba94d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cb032b45-8976-4ea6-9b05-1dbb6e6ba94d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:35:55.416529 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.416496 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cb032b45-8976-4ea6-9b05-1dbb6e6ba94d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cb032b45-8976-4ea6-9b05-1dbb6e6ba94d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:35:55.416529 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.416518 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cb032b45-8976-4ea6-9b05-1dbb6e6ba94d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cb032b45-8976-4ea6-9b05-1dbb6e6ba94d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:35:55.416894 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.416548 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwmsb\" (UniqueName: \"kubernetes.io/projected/cb032b45-8976-4ea6-9b05-1dbb6e6ba94d-kube-api-access-xwmsb\") pod \"prometheus-k8s-0\" (UID: \"cb032b45-8976-4ea6-9b05-1dbb6e6ba94d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:35:55.416894 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.416578 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cb032b45-8976-4ea6-9b05-1dbb6e6ba94d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cb032b45-8976-4ea6-9b05-1dbb6e6ba94d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:35:55.416894 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.416606 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cb032b45-8976-4ea6-9b05-1dbb6e6ba94d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cb032b45-8976-4ea6-9b05-1dbb6e6ba94d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:35:55.416894 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.416645 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cb032b45-8976-4ea6-9b05-1dbb6e6ba94d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cb032b45-8976-4ea6-9b05-1dbb6e6ba94d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:35:55.416894 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.416669 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cb032b45-8976-4ea6-9b05-1dbb6e6ba94d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cb032b45-8976-4ea6-9b05-1dbb6e6ba94d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:35:55.416894 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.416698 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb032b45-8976-4ea6-9b05-1dbb6e6ba94d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cb032b45-8976-4ea6-9b05-1dbb6e6ba94d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:35:55.416894 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.416725 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cb032b45-8976-4ea6-9b05-1dbb6e6ba94d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cb032b45-8976-4ea6-9b05-1dbb6e6ba94d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:35:55.416894 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.416811 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cb032b45-8976-4ea6-9b05-1dbb6e6ba94d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cb032b45-8976-4ea6-9b05-1dbb6e6ba94d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:35:55.417751 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.417605 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb032b45-8976-4ea6-9b05-1dbb6e6ba94d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cb032b45-8976-4ea6-9b05-1dbb6e6ba94d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:35:55.417751 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.417644 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cb032b45-8976-4ea6-9b05-1dbb6e6ba94d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cb032b45-8976-4ea6-9b05-1dbb6e6ba94d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:35:55.419737 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.418261 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb032b45-8976-4ea6-9b05-1dbb6e6ba94d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cb032b45-8976-4ea6-9b05-1dbb6e6ba94d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:35:55.419737 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.418888 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cb032b45-8976-4ea6-9b05-1dbb6e6ba94d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cb032b45-8976-4ea6-9b05-1dbb6e6ba94d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:35:55.419737 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.419456 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cb032b45-8976-4ea6-9b05-1dbb6e6ba94d-config-out\") pod \"prometheus-k8s-0\" (UID: \"cb032b45-8976-4ea6-9b05-1dbb6e6ba94d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:35:55.419737 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.419481 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cb032b45-8976-4ea6-9b05-1dbb6e6ba94d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cb032b45-8976-4ea6-9b05-1dbb6e6ba94d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:35:55.420395 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.419805 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cb032b45-8976-4ea6-9b05-1dbb6e6ba94d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cb032b45-8976-4ea6-9b05-1dbb6e6ba94d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:35:55.420395 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.419887 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cb032b45-8976-4ea6-9b05-1dbb6e6ba94d-web-config\") pod \"prometheus-k8s-0\" (UID: \"cb032b45-8976-4ea6-9b05-1dbb6e6ba94d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:35:55.420395 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.420330 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb032b45-8976-4ea6-9b05-1dbb6e6ba94d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cb032b45-8976-4ea6-9b05-1dbb6e6ba94d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:35:55.420395 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.420376 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cb032b45-8976-4ea6-9b05-1dbb6e6ba94d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cb032b45-8976-4ea6-9b05-1dbb6e6ba94d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:35:55.420395 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.420389 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/cb032b45-8976-4ea6-9b05-1dbb6e6ba94d-config\") pod \"prometheus-k8s-0\" (UID: \"cb032b45-8976-4ea6-9b05-1dbb6e6ba94d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:35:55.420718 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.420693 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cb032b45-8976-4ea6-9b05-1dbb6e6ba94d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cb032b45-8976-4ea6-9b05-1dbb6e6ba94d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:35:55.421375 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.421354 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cb032b45-8976-4ea6-9b05-1dbb6e6ba94d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cb032b45-8976-4ea6-9b05-1dbb6e6ba94d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:35:55.422098 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.422079 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cb032b45-8976-4ea6-9b05-1dbb6e6ba94d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cb032b45-8976-4ea6-9b05-1dbb6e6ba94d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:35:55.422276 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.422258 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cb032b45-8976-4ea6-9b05-1dbb6e6ba94d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cb032b45-8976-4ea6-9b05-1dbb6e6ba94d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:35:55.422428 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.422409 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cb032b45-8976-4ea6-9b05-1dbb6e6ba94d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cb032b45-8976-4ea6-9b05-1dbb6e6ba94d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:35:55.422529 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.422509 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cb032b45-8976-4ea6-9b05-1dbb6e6ba94d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cb032b45-8976-4ea6-9b05-1dbb6e6ba94d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:35:55.422662 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.422643 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cb032b45-8976-4ea6-9b05-1dbb6e6ba94d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cb032b45-8976-4ea6-9b05-1dbb6e6ba94d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:35:55.425876 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.425858 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwmsb\" (UniqueName: \"kubernetes.io/projected/cb032b45-8976-4ea6-9b05-1dbb6e6ba94d-kube-api-access-xwmsb\") pod \"prometheus-k8s-0\" (UID: \"cb032b45-8976-4ea6-9b05-1dbb6e6ba94d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:35:55.442251 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.442226 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5295084b-def6-431a-b103-b82d9b3b7f74" path="/var/lib/kubelet/pods/5295084b-def6-431a-b103-b82d9b3b7f74/volumes" Jul 09 09:35:55.550430 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.550409 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:35:55.683280 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:55.683256 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 09:35:55.685368 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:35:55.685348 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb032b45_8976_4ea6_9b05_1dbb6e6ba94d.slice/crio-afb5b136eb00bb42a10c596c063499a2197184e418c75768ae36eee7b166f4f7 WatchSource:0}: Error finding container afb5b136eb00bb42a10c596c063499a2197184e418c75768ae36eee7b166f4f7: Status 404 returned error can't find the container with id afb5b136eb00bb42a10c596c063499a2197184e418c75768ae36eee7b166f4f7 Jul 09 09:35:56.178741 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:56.178707 2578 generic.go:356] "Generic (PLEG): container finished" podID="cb032b45-8976-4ea6-9b05-1dbb6e6ba94d" containerID="5db087f1ba6ea6e99697f3eac6a368e3144597a1d82ebab85bfab619238b94b8" exitCode=0 Jul 09 09:35:56.179142 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:56.178797 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cb032b45-8976-4ea6-9b05-1dbb6e6ba94d","Type":"ContainerDied","Data":"5db087f1ba6ea6e99697f3eac6a368e3144597a1d82ebab85bfab619238b94b8"} Jul 09 09:35:56.179142 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:56.178906 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cb032b45-8976-4ea6-9b05-1dbb6e6ba94d","Type":"ContainerStarted","Data":"afb5b136eb00bb42a10c596c063499a2197184e418c75768ae36eee7b166f4f7"} Jul 09 09:35:57.186310 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:57.186280 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cb032b45-8976-4ea6-9b05-1dbb6e6ba94d","Type":"ContainerStarted","Data":"4510aa221e952b1d102f9cab383d25009dc0e5689ba4ece3abf7db130bdc71c8"} Jul 09 09:35:57.186684 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:57.186341 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cb032b45-8976-4ea6-9b05-1dbb6e6ba94d","Type":"ContainerStarted","Data":"67e8d334a148b9c8a5ddac715e4544147b07e72beea1ada626efcbf29d35f816"} Jul 09 09:35:57.186684 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:57.186367 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cb032b45-8976-4ea6-9b05-1dbb6e6ba94d","Type":"ContainerStarted","Data":"a9b3430c04993686e3988d183adb7a166ad88989ab7e55b11f7b4d52f19728f9"} Jul 09 09:35:57.186684 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:57.186391 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cb032b45-8976-4ea6-9b05-1dbb6e6ba94d","Type":"ContainerStarted","Data":"50ae582714d026969d261d55c95122495b84981e6f03b382623d59a480e235a3"} Jul 09 09:35:57.186684 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:57.186415 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cb032b45-8976-4ea6-9b05-1dbb6e6ba94d","Type":"ContainerStarted","Data":"dbfe3ac3522e4e1e80c1d3e5c56c7aa8fd3606475ab731f57ce08a1c0884028d"} Jul 09 09:35:57.186684 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:57.186437 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cb032b45-8976-4ea6-9b05-1dbb6e6ba94d","Type":"ContainerStarted","Data":"8bdc5906e18c14847bb13f127fdb4a387029c00c028a29ca3247fb8000caae46"} Jul 09 09:35:57.216397 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:35:57.216353 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.197402498 podStartE2EDuration="2.216337345s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.934847ms" isStatefulPod=false podCreationTimestamp="2026-07-09 09:35:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:35:57.214353267 +0000 UTC m=+142.306107188" watchObservedRunningTime="2026-07-09 09:35:57.216337345 +0000 UTC m=+142.308091265" Jul 09 09:36:00.551449 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:36:00.551414 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:36:55.550979 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:36:55.550944 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:36:55.566798 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:36:55.566771 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:36:56.409419 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:36:56.409389 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 09:38:35.306259 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:38:35.306228 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xf4sw_1677420d-7376-45ad-ac67-20128aec5756/ovn-acl-logging/0.log" Jul 09 09:38:35.310820 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:38:35.310797 2578 kubelet.go:1648] "Image garbage collection succeeded" Jul 09 09:38:35.311588 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:38:35.311567 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xf4sw_1677420d-7376-45ad-ac67-20128aec5756/ovn-acl-logging/0.log" Jul 09 09:39:12.337725 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:39:12.337695 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-68bff59c-g7djw"] Jul 09 09:39:12.339724 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:39:12.339707 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-68bff59c-g7djw" Jul 09 09:39:12.342445 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:39:12.342423 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\"" Jul 09 09:39:12.342580 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:39:12.342485 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\"" Jul 09 09:39:12.342580 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:39:12.342522 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\"" Jul 09 09:39:12.343399 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:39:12.343384 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\"" Jul 09 09:39:12.343505 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:39:12.343384 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\"" Jul 09 09:39:12.343505 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:39:12.343406 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-x297v\"" Jul 09 09:39:12.349988 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:39:12.349967 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-68bff59c-g7djw"] Jul 09 09:39:12.417501 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:39:12.417473 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3cd5cac2-f5a4-468b-8633-df1425ccef3e-certificates\") pod \"keda-operator-68bff59c-g7djw\" (UID: \"3cd5cac2-f5a4-468b-8633-df1425ccef3e\") " pod="openshift-keda/keda-operator-68bff59c-g7djw" Jul 09 09:39:12.417596 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:39:12.417534 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/3cd5cac2-f5a4-468b-8633-df1425ccef3e-cabundle0\") pod \"keda-operator-68bff59c-g7djw\" (UID: \"3cd5cac2-f5a4-468b-8633-df1425ccef3e\") " pod="openshift-keda/keda-operator-68bff59c-g7djw" Jul 09 09:39:12.417596 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:39:12.417559 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqf4z\" (UniqueName: \"kubernetes.io/projected/3cd5cac2-f5a4-468b-8633-df1425ccef3e-kube-api-access-vqf4z\") pod \"keda-operator-68bff59c-g7djw\" (UID: \"3cd5cac2-f5a4-468b-8633-df1425ccef3e\") " pod="openshift-keda/keda-operator-68bff59c-g7djw" Jul 09 09:39:12.518430 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:39:12.518400 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3cd5cac2-f5a4-468b-8633-df1425ccef3e-certificates\") pod \"keda-operator-68bff59c-g7djw\" (UID: \"3cd5cac2-f5a4-468b-8633-df1425ccef3e\") " pod="openshift-keda/keda-operator-68bff59c-g7djw" Jul 09 09:39:12.518540 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:39:12.518448 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/3cd5cac2-f5a4-468b-8633-df1425ccef3e-cabundle0\") pod \"keda-operator-68bff59c-g7djw\" (UID: \"3cd5cac2-f5a4-468b-8633-df1425ccef3e\") " pod="openshift-keda/keda-operator-68bff59c-g7djw" Jul 09 09:39:12.518540 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:39:12.518466 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqf4z\" (UniqueName: \"kubernetes.io/projected/3cd5cac2-f5a4-468b-8633-df1425ccef3e-kube-api-access-vqf4z\") pod \"keda-operator-68bff59c-g7djw\" (UID: \"3cd5cac2-f5a4-468b-8633-df1425ccef3e\") " pod="openshift-keda/keda-operator-68bff59c-g7djw" Jul 09 09:39:12.518621 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:39:12.518540 2578 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found Jul 09 09:39:12.518621 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:39:12.518558 2578 secret.go:281] references non-existent secret key: ca.crt Jul 09 09:39:12.518621 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:39:12.518568 2578 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt Jul 09 09:39:12.518621 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:39:12.518579 2578 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-68bff59c-g7djw: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt] Jul 09 09:39:12.518758 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:39:12.518626 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3cd5cac2-f5a4-468b-8633-df1425ccef3e-certificates podName:3cd5cac2-f5a4-468b-8633-df1425ccef3e nodeName:}" failed. No retries permitted until 2026-07-09 09:39:13.018612704 +0000 UTC m=+338.110366601 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/3cd5cac2-f5a4-468b-8633-df1425ccef3e-certificates") pod "keda-operator-68bff59c-g7djw" (UID: "3cd5cac2-f5a4-468b-8633-df1425ccef3e") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt] Jul 09 09:39:12.519028 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:39:12.519011 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/3cd5cac2-f5a4-468b-8633-df1425ccef3e-cabundle0\") pod \"keda-operator-68bff59c-g7djw\" (UID: \"3cd5cac2-f5a4-468b-8633-df1425ccef3e\") " pod="openshift-keda/keda-operator-68bff59c-g7djw" Jul 09 09:39:12.527524 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:39:12.527508 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqf4z\" (UniqueName: \"kubernetes.io/projected/3cd5cac2-f5a4-468b-8633-df1425ccef3e-kube-api-access-vqf4z\") pod \"keda-operator-68bff59c-g7djw\" (UID: \"3cd5cac2-f5a4-468b-8633-df1425ccef3e\") " pod="openshift-keda/keda-operator-68bff59c-g7djw" Jul 09 09:39:12.683332 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:39:12.683306 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-sxv77"] Jul 09 09:39:12.685667 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:39:12.685649 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-559f75f947-sxv77" Jul 09 09:39:12.688207 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:39:12.688187 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\"" Jul 09 09:39:12.695800 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:39:12.695778 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-sxv77"] Jul 09 09:39:12.800537 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:39:12.800510 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-gbrb9"] Jul 09 09:39:12.802647 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:39:12.802629 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-5fbd5c4644-gbrb9" Jul 09 09:39:12.804900 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:39:12.804884 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\"" Jul 09 09:39:12.813542 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:39:12.813522 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-gbrb9"] Jul 09 09:39:12.821566 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:39:12.821541 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/3dcc4ca8-a0ac-4f06-8d6c-c7944d57384e-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-sxv77\" (UID: \"3dcc4ca8-a0ac-4f06-8d6c-c7944d57384e\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-sxv77" Jul 09 09:39:12.821665 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:39:12.821645 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnq7p\" (UniqueName: \"kubernetes.io/projected/3dcc4ca8-a0ac-4f06-8d6c-c7944d57384e-kube-api-access-rnq7p\") pod \"keda-metrics-apiserver-559f75f947-sxv77\" (UID: \"3dcc4ca8-a0ac-4f06-8d6c-c7944d57384e\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-sxv77" Jul 09 09:39:12.821762 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:39:12.821741 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3dcc4ca8-a0ac-4f06-8d6c-c7944d57384e-certificates\") pod \"keda-metrics-apiserver-559f75f947-sxv77\" (UID: \"3dcc4ca8-a0ac-4f06-8d6c-c7944d57384e\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-sxv77" Jul 09 09:39:12.922331 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:39:12.922291 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnq7p\" (UniqueName: \"kubernetes.io/projected/3dcc4ca8-a0ac-4f06-8d6c-c7944d57384e-kube-api-access-rnq7p\") pod \"keda-metrics-apiserver-559f75f947-sxv77\" (UID: \"3dcc4ca8-a0ac-4f06-8d6c-c7944d57384e\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-sxv77" Jul 09 09:39:12.923176 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:39:12.922572 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3dcc4ca8-a0ac-4f06-8d6c-c7944d57384e-certificates\") pod \"keda-metrics-apiserver-559f75f947-sxv77\" (UID: \"3dcc4ca8-a0ac-4f06-8d6c-c7944d57384e\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-sxv77" Jul 09 09:39:12.923176 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:39:12.922613 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4fe0e50b-0876-45c9-88d8-71eb32f83a9e-certificates\") pod \"keda-admission-5fbd5c4644-gbrb9\" (UID: \"4fe0e50b-0876-45c9-88d8-71eb32f83a9e\") " pod="openshift-keda/keda-admission-5fbd5c4644-gbrb9" Jul 09 09:39:12.923176 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:39:12.922653 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/3dcc4ca8-a0ac-4f06-8d6c-c7944d57384e-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-sxv77\" (UID: \"3dcc4ca8-a0ac-4f06-8d6c-c7944d57384e\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-sxv77" Jul 09 09:39:12.923176 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:39:12.922681 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g662k\" (UniqueName: \"kubernetes.io/projected/4fe0e50b-0876-45c9-88d8-71eb32f83a9e-kube-api-access-g662k\") pod \"keda-admission-5fbd5c4644-gbrb9\" (UID: \"4fe0e50b-0876-45c9-88d8-71eb32f83a9e\") " pod="openshift-keda/keda-admission-5fbd5c4644-gbrb9" Jul 09 09:39:12.923176 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:39:12.922950 2578 projected.go:266] Couldn't get secret openshift-keda/keda-metrics-apiserver-certs: secret "keda-metrics-apiserver-certs" not found Jul 09 09:39:12.923176 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:39:12.922973 2578 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-559f75f947-sxv77: secret "keda-metrics-apiserver-certs" not found Jul 09 09:39:12.923176 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:39:12.923048 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3dcc4ca8-a0ac-4f06-8d6c-c7944d57384e-certificates podName:3dcc4ca8-a0ac-4f06-8d6c-c7944d57384e nodeName:}" failed. No retries permitted until 2026-07-09 09:39:13.423030401 +0000 UTC m=+338.514784303 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/3dcc4ca8-a0ac-4f06-8d6c-c7944d57384e-certificates") pod "keda-metrics-apiserver-559f75f947-sxv77" (UID: "3dcc4ca8-a0ac-4f06-8d6c-c7944d57384e") : secret "keda-metrics-apiserver-certs" not found Jul 09 09:39:12.923176 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:39:12.923133 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/3dcc4ca8-a0ac-4f06-8d6c-c7944d57384e-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-sxv77\" (UID: \"3dcc4ca8-a0ac-4f06-8d6c-c7944d57384e\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-sxv77" Jul 09 09:39:12.932088 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:39:12.932064 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnq7p\" (UniqueName: \"kubernetes.io/projected/3dcc4ca8-a0ac-4f06-8d6c-c7944d57384e-kube-api-access-rnq7p\") pod \"keda-metrics-apiserver-559f75f947-sxv77\" (UID: \"3dcc4ca8-a0ac-4f06-8d6c-c7944d57384e\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-sxv77" Jul 09 09:39:13.023504 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:39:13.023422 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4fe0e50b-0876-45c9-88d8-71eb32f83a9e-certificates\") pod \"keda-admission-5fbd5c4644-gbrb9\" (UID: \"4fe0e50b-0876-45c9-88d8-71eb32f83a9e\") " pod="openshift-keda/keda-admission-5fbd5c4644-gbrb9" Jul 09 09:39:13.023504 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:39:13.023493 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g662k\" (UniqueName: \"kubernetes.io/projected/4fe0e50b-0876-45c9-88d8-71eb32f83a9e-kube-api-access-g662k\") pod \"keda-admission-5fbd5c4644-gbrb9\" (UID: \"4fe0e50b-0876-45c9-88d8-71eb32f83a9e\") " pod="openshift-keda/keda-admission-5fbd5c4644-gbrb9" Jul 09 09:39:13.023695 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:39:13.023521 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3cd5cac2-f5a4-468b-8633-df1425ccef3e-certificates\") pod \"keda-operator-68bff59c-g7djw\" (UID: \"3cd5cac2-f5a4-468b-8633-df1425ccef3e\") " pod="openshift-keda/keda-operator-68bff59c-g7djw" Jul 09 09:39:13.023955 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:39:13.023925 2578 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found Jul 09 09:39:13.024050 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:39:13.023960 2578 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-5fbd5c4644-gbrb9: secret "keda-admission-webhooks-certs" not found Jul 09 09:39:13.024050 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:39:13.024014 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4fe0e50b-0876-45c9-88d8-71eb32f83a9e-certificates podName:4fe0e50b-0876-45c9-88d8-71eb32f83a9e nodeName:}" failed. No retries permitted until 2026-07-09 09:39:13.524000885 +0000 UTC m=+338.615754787 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/4fe0e50b-0876-45c9-88d8-71eb32f83a9e-certificates") pod "keda-admission-5fbd5c4644-gbrb9" (UID: "4fe0e50b-0876-45c9-88d8-71eb32f83a9e") : secret "keda-admission-webhooks-certs" not found Jul 09 09:39:13.026300 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:39:13.026281 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3cd5cac2-f5a4-468b-8633-df1425ccef3e-certificates\") pod \"keda-operator-68bff59c-g7djw\" (UID: \"3cd5cac2-f5a4-468b-8633-df1425ccef3e\") " pod="openshift-keda/keda-operator-68bff59c-g7djw" Jul 09 09:39:13.034242 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:39:13.034223 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g662k\" (UniqueName: \"kubernetes.io/projected/4fe0e50b-0876-45c9-88d8-71eb32f83a9e-kube-api-access-g662k\") pod \"keda-admission-5fbd5c4644-gbrb9\" (UID: \"4fe0e50b-0876-45c9-88d8-71eb32f83a9e\") " pod="openshift-keda/keda-admission-5fbd5c4644-gbrb9" Jul 09 09:39:13.249925 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:39:13.249888 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-68bff59c-g7djw" Jul 09 09:39:13.386708 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:39:13.386678 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-68bff59c-g7djw"] Jul 09 09:39:13.390310 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:39:13.390279 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cd5cac2_f5a4_468b_8633_df1425ccef3e.slice/crio-e044b0aaac62d3e27b22ce6767502ae09af9b4b9e02c638eebe68f955d03edc6 WatchSource:0}: Error finding container e044b0aaac62d3e27b22ce6767502ae09af9b4b9e02c638eebe68f955d03edc6: Status 404 returned error can't find the container with id e044b0aaac62d3e27b22ce6767502ae09af9b4b9e02c638eebe68f955d03edc6 Jul 09 09:39:13.391642 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:39:13.391626 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 09:39:13.426627 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:39:13.426602 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3dcc4ca8-a0ac-4f06-8d6c-c7944d57384e-certificates\") pod \"keda-metrics-apiserver-559f75f947-sxv77\" (UID: \"3dcc4ca8-a0ac-4f06-8d6c-c7944d57384e\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-sxv77" Jul 09 09:39:13.429060 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:39:13.429044 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3dcc4ca8-a0ac-4f06-8d6c-c7944d57384e-certificates\") pod \"keda-metrics-apiserver-559f75f947-sxv77\" (UID: \"3dcc4ca8-a0ac-4f06-8d6c-c7944d57384e\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-sxv77" Jul 09 09:39:13.527158 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:39:13.527131 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4fe0e50b-0876-45c9-88d8-71eb32f83a9e-certificates\") pod \"keda-admission-5fbd5c4644-gbrb9\" (UID: \"4fe0e50b-0876-45c9-88d8-71eb32f83a9e\") " pod="openshift-keda/keda-admission-5fbd5c4644-gbrb9" Jul 09 09:39:13.527269 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:39:13.527257 2578 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found Jul 09 09:39:13.527311 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:39:13.527277 2578 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-5fbd5c4644-gbrb9: secret "keda-admission-webhooks-certs" not found Jul 09 09:39:13.527344 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:39:13.527319 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4fe0e50b-0876-45c9-88d8-71eb32f83a9e-certificates podName:4fe0e50b-0876-45c9-88d8-71eb32f83a9e nodeName:}" failed. No retries permitted until 2026-07-09 09:39:14.527306931 +0000 UTC m=+339.619060829 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/4fe0e50b-0876-45c9-88d8-71eb32f83a9e-certificates") pod "keda-admission-5fbd5c4644-gbrb9" (UID: "4fe0e50b-0876-45c9-88d8-71eb32f83a9e") : secret "keda-admission-webhooks-certs" not found Jul 09 09:39:13.595423 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:39:13.595371 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-559f75f947-sxv77" Jul 09 09:39:13.716262 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:39:13.716231 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-sxv77"] Jul 09 09:39:13.719135 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:39:13.719109 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dcc4ca8_a0ac_4f06_8d6c_c7944d57384e.slice/crio-e004f31553ca042c5879a0712d501d3bb3ff45d5d7225cf8ab097b2db5033b8f WatchSource:0}: Error finding container e004f31553ca042c5879a0712d501d3bb3ff45d5d7225cf8ab097b2db5033b8f: Status 404 returned error can't find the container with id e004f31553ca042c5879a0712d501d3bb3ff45d5d7225cf8ab097b2db5033b8f Jul 09 09:39:13.846990 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:39:13.846924 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-559f75f947-sxv77" event={"ID":"3dcc4ca8-a0ac-4f06-8d6c-c7944d57384e","Type":"ContainerStarted","Data":"e004f31553ca042c5879a0712d501d3bb3ff45d5d7225cf8ab097b2db5033b8f"} Jul 09 09:39:13.848014 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:39:13.847992 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-68bff59c-g7djw" event={"ID":"3cd5cac2-f5a4-468b-8633-df1425ccef3e","Type":"ContainerStarted","Data":"e044b0aaac62d3e27b22ce6767502ae09af9b4b9e02c638eebe68f955d03edc6"} Jul 09 09:39:14.536348 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:39:14.536311 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4fe0e50b-0876-45c9-88d8-71eb32f83a9e-certificates\") pod \"keda-admission-5fbd5c4644-gbrb9\" (UID: \"4fe0e50b-0876-45c9-88d8-71eb32f83a9e\") " pod="openshift-keda/keda-admission-5fbd5c4644-gbrb9" Jul 09 09:39:14.540137 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:39:14.540105 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4fe0e50b-0876-45c9-88d8-71eb32f83a9e-certificates\") pod \"keda-admission-5fbd5c4644-gbrb9\" (UID: \"4fe0e50b-0876-45c9-88d8-71eb32f83a9e\") " pod="openshift-keda/keda-admission-5fbd5c4644-gbrb9" Jul 09 09:39:14.612234 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:39:14.612202 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-5fbd5c4644-gbrb9" Jul 09 09:39:14.794449 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:39:14.794326 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-gbrb9"] Jul 09 09:39:14.800005 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:39:14.799931 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fe0e50b_0876_45c9_88d8_71eb32f83a9e.slice/crio-faa398711953bb050af52c2caeabb3f729f3d3f74537dae3aca7e3b5ac10f363 WatchSource:0}: Error finding container faa398711953bb050af52c2caeabb3f729f3d3f74537dae3aca7e3b5ac10f363: Status 404 returned error can't find the container with id faa398711953bb050af52c2caeabb3f729f3d3f74537dae3aca7e3b5ac10f363 Jul 09 09:39:14.854487 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:39:14.854455 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-5fbd5c4644-gbrb9" event={"ID":"4fe0e50b-0876-45c9-88d8-71eb32f83a9e","Type":"ContainerStarted","Data":"faa398711953bb050af52c2caeabb3f729f3d3f74537dae3aca7e3b5ac10f363"} Jul 09 09:39:18.873260 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:39:18.873220 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-559f75f947-sxv77" event={"ID":"3dcc4ca8-a0ac-4f06-8d6c-c7944d57384e","Type":"ContainerStarted","Data":"6e28c25d7ffa31d27e05335b744edb3525c702da37508b3fe10d9a04e67d2c3f"} Jul 09 09:39:18.873699 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:39:18.873461 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-559f75f947-sxv77" Jul 09 09:39:18.874815 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:39:18.874793 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-68bff59c-g7djw" event={"ID":"3cd5cac2-f5a4-468b-8633-df1425ccef3e","Type":"ContainerStarted","Data":"10ac044df2009935ec74e0779224257e513629062122f39502528099dcab6906"} Jul 09 09:39:18.874912 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:39:18.874867 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-68bff59c-g7djw" Jul 09 09:39:18.876203 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:39:18.876180 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-5fbd5c4644-gbrb9" event={"ID":"4fe0e50b-0876-45c9-88d8-71eb32f83a9e","Type":"ContainerStarted","Data":"e6ec117e2b3505136fe2794eb342cec9eef943d72fa4f45164ee4ad5e1a9cd66"} Jul 09 09:39:18.876337 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:39:18.876305 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-5fbd5c4644-gbrb9" Jul 09 09:39:18.890950 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:39:18.890906 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-559f75f947-sxv77" podStartSLOduration=2.307464657 podStartE2EDuration="6.890895328s" totalImagesPullingTime="4.583430671s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:39:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:39:18.889221437 +0000 UTC m=+343.980975353" watchObservedRunningTime="2026-07-09 09:39:18.890895328 +0000 UTC m=+343.982649249" Jul 09 09:39:18.905510 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:39:18.905465 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-68bff59c-g7djw" podStartSLOduration=2.023027865 podStartE2EDuration="6.905453878s" totalImagesPullingTime="4.882426013s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:39:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:39:18.904650119 +0000 UTC m=+343.996404040" watchObservedRunningTime="2026-07-09 09:39:18.905453878 +0000 UTC m=+343.997207797" Jul 09 09:39:18.920355 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:39:18.920311 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-5fbd5c4644-gbrb9" podStartSLOduration=3.417185866 podStartE2EDuration="6.920298893s" totalImagesPullingTime="3.503113027s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:39:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:39:18.920006939 +0000 UTC m=+344.011760859" watchObservedRunningTime="2026-07-09 09:39:18.920298893 +0000 UTC m=+344.012052813" Jul 09 09:39:29.884742 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:39:29.884662 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-559f75f947-sxv77" Jul 09 09:39:39.882613 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:39:39.882579 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-68bff59c-g7djw" Jul 09 09:39:39.883072 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:39:39.882803 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-5fbd5c4644-gbrb9" Jul 09 09:40:21.085794 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:40:21.085763 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-7bqkz"] Jul 09 09:40:21.088059 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:40:21.088042 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-7bqkz" Jul 09 09:40:21.090552 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:40:21.090533 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" Jul 09 09:40:21.090657 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:40:21.090642 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-pkw2j\"" Jul 09 09:40:21.091462 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:40:21.091441 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\"" Jul 09 09:40:21.091617 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:40:21.091468 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" Jul 09 09:40:21.098397 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:40:21.098378 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-7bqkz"] Jul 09 09:40:21.236349 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:40:21.236322 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgxk6\" (UniqueName: \"kubernetes.io/projected/e8001e12-b4b6-4781-8210-b4e0be833a53-kube-api-access-wgxk6\") pod \"seaweedfs-64568bcd49-7bqkz\" (UID: \"e8001e12-b4b6-4781-8210-b4e0be833a53\") " pod="kserve/seaweedfs-64568bcd49-7bqkz" Jul 09 09:40:21.236517 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:40:21.236378 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e8001e12-b4b6-4781-8210-b4e0be833a53-data\") pod \"seaweedfs-64568bcd49-7bqkz\" (UID: \"e8001e12-b4b6-4781-8210-b4e0be833a53\") " pod="kserve/seaweedfs-64568bcd49-7bqkz" Jul 09 09:40:21.337062 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:40:21.336972 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgxk6\" (UniqueName: \"kubernetes.io/projected/e8001e12-b4b6-4781-8210-b4e0be833a53-kube-api-access-wgxk6\") pod \"seaweedfs-64568bcd49-7bqkz\" (UID: \"e8001e12-b4b6-4781-8210-b4e0be833a53\") " pod="kserve/seaweedfs-64568bcd49-7bqkz" Jul 09 09:40:21.337194 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:40:21.337071 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e8001e12-b4b6-4781-8210-b4e0be833a53-data\") pod \"seaweedfs-64568bcd49-7bqkz\" (UID: \"e8001e12-b4b6-4781-8210-b4e0be833a53\") " pod="kserve/seaweedfs-64568bcd49-7bqkz" Jul 09 09:40:21.337441 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:40:21.337421 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e8001e12-b4b6-4781-8210-b4e0be833a53-data\") pod \"seaweedfs-64568bcd49-7bqkz\" (UID: \"e8001e12-b4b6-4781-8210-b4e0be833a53\") " pod="kserve/seaweedfs-64568bcd49-7bqkz" Jul 09 09:40:21.344943 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:40:21.344918 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgxk6\" (UniqueName: \"kubernetes.io/projected/e8001e12-b4b6-4781-8210-b4e0be833a53-kube-api-access-wgxk6\") pod \"seaweedfs-64568bcd49-7bqkz\" (UID: \"e8001e12-b4b6-4781-8210-b4e0be833a53\") " pod="kserve/seaweedfs-64568bcd49-7bqkz" Jul 09 09:40:21.397945 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:40:21.397922 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-7bqkz" Jul 09 09:40:21.523652 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:40:21.523622 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-7bqkz"] Jul 09 09:40:21.526775 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:40:21.526750 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8001e12_b4b6_4781_8210_b4e0be833a53.slice/crio-983f1dc2a8cf20c432ac94dfd1cdb75abe9473392cd8ed208eb473a823974984 WatchSource:0}: Error finding container 983f1dc2a8cf20c432ac94dfd1cdb75abe9473392cd8ed208eb473a823974984: Status 404 returned error can't find the container with id 983f1dc2a8cf20c432ac94dfd1cdb75abe9473392cd8ed208eb473a823974984 Jul 09 09:40:22.105050 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:40:22.105018 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-7bqkz" event={"ID":"e8001e12-b4b6-4781-8210-b4e0be833a53","Type":"ContainerStarted","Data":"983f1dc2a8cf20c432ac94dfd1cdb75abe9473392cd8ed208eb473a823974984"} Jul 09 09:40:27.125333 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:40:27.125298 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-7bqkz" event={"ID":"e8001e12-b4b6-4781-8210-b4e0be833a53","Type":"ContainerStarted","Data":"22218c9934a63ff0c7f6a8b76a462711a01883c33bf81f5cdb6ff32afe15c255"} Jul 09 09:40:27.125805 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:40:27.125434 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-7bqkz" Jul 09 09:40:27.146630 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:40:27.146581 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-7bqkz" podStartSLOduration=1.484523333 podStartE2EDuration="6.146566916s" totalImagesPullingTime="4.662043583s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:40:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:40:27.144163274 +0000 UTC m=+412.235917194" watchObservedRunningTime="2026-07-09 09:40:27.146566916 +0000 UTC m=+412.238320835" Jul 09 09:40:33.131630 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:40:33.131594 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-7bqkz" Jul 09 09:41:14.745926 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:14.745824 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-85f5448947-c6q9r"] Jul 09 09:41:14.750074 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:14.750058 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85f5448947-c6q9r" Jul 09 09:41:14.755246 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:14.755218 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 09 09:41:14.756045 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:14.756022 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 09 09:41:14.756210 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:14.756028 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 09 09:41:14.756620 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:14.756212 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-5sxl6\"" Jul 09 09:41:14.757674 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:14.757652 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 09 09:41:14.757924 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:14.757904 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 09 09:41:14.777635 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:14.777613 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 09 09:41:14.777976 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:14.777960 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85f5448947-c6q9r"] Jul 09 09:41:14.842270 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:14.842245 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9f4cb38c-f6ba-4250-a710-a465a6d1d74c-console-oauth-config\") pod \"console-85f5448947-c6q9r\" (UID: \"9f4cb38c-f6ba-4250-a710-a465a6d1d74c\") " pod="openshift-console/console-85f5448947-c6q9r" Jul 09 09:41:14.842442 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:14.842283 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9f4cb38c-f6ba-4250-a710-a465a6d1d74c-console-serving-cert\") pod \"console-85f5448947-c6q9r\" (UID: \"9f4cb38c-f6ba-4250-a710-a465a6d1d74c\") " pod="openshift-console/console-85f5448947-c6q9r" Jul 09 09:41:14.842442 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:14.842303 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f4cb38c-f6ba-4250-a710-a465a6d1d74c-trusted-ca-bundle\") pod \"console-85f5448947-c6q9r\" (UID: \"9f4cb38c-f6ba-4250-a710-a465a6d1d74c\") " pod="openshift-console/console-85f5448947-c6q9r" Jul 09 09:41:14.842442 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:14.842320 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9f4cb38c-f6ba-4250-a710-a465a6d1d74c-oauth-serving-cert\") pod \"console-85f5448947-c6q9r\" (UID: \"9f4cb38c-f6ba-4250-a710-a465a6d1d74c\") " pod="openshift-console/console-85f5448947-c6q9r" Jul 09 09:41:14.842442 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:14.842365 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tknkq\" (UniqueName: \"kubernetes.io/projected/9f4cb38c-f6ba-4250-a710-a465a6d1d74c-kube-api-access-tknkq\") pod \"console-85f5448947-c6q9r\" (UID: \"9f4cb38c-f6ba-4250-a710-a465a6d1d74c\") " pod="openshift-console/console-85f5448947-c6q9r" Jul 09 09:41:14.842442 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:14.842399 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9f4cb38c-f6ba-4250-a710-a465a6d1d74c-console-config\") pod \"console-85f5448947-c6q9r\" (UID: \"9f4cb38c-f6ba-4250-a710-a465a6d1d74c\") " pod="openshift-console/console-85f5448947-c6q9r" Jul 09 09:41:14.842442 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:14.842415 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9f4cb38c-f6ba-4250-a710-a465a6d1d74c-service-ca\") pod \"console-85f5448947-c6q9r\" (UID: \"9f4cb38c-f6ba-4250-a710-a465a6d1d74c\") " pod="openshift-console/console-85f5448947-c6q9r" Jul 09 09:41:14.943271 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:14.943240 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9f4cb38c-f6ba-4250-a710-a465a6d1d74c-console-config\") pod \"console-85f5448947-c6q9r\" (UID: \"9f4cb38c-f6ba-4250-a710-a465a6d1d74c\") " pod="openshift-console/console-85f5448947-c6q9r" Jul 09 09:41:14.943271 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:14.943269 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9f4cb38c-f6ba-4250-a710-a465a6d1d74c-service-ca\") pod \"console-85f5448947-c6q9r\" (UID: \"9f4cb38c-f6ba-4250-a710-a465a6d1d74c\") " pod="openshift-console/console-85f5448947-c6q9r" Jul 09 09:41:14.943486 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:14.943301 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9f4cb38c-f6ba-4250-a710-a465a6d1d74c-console-oauth-config\") pod \"console-85f5448947-c6q9r\" (UID: \"9f4cb38c-f6ba-4250-a710-a465a6d1d74c\") " pod="openshift-console/console-85f5448947-c6q9r" Jul 09 09:41:14.943486 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:14.943328 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9f4cb38c-f6ba-4250-a710-a465a6d1d74c-console-serving-cert\") pod \"console-85f5448947-c6q9r\" (UID: \"9f4cb38c-f6ba-4250-a710-a465a6d1d74c\") " pod="openshift-console/console-85f5448947-c6q9r" Jul 09 09:41:14.943486 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:14.943344 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f4cb38c-f6ba-4250-a710-a465a6d1d74c-trusted-ca-bundle\") pod \"console-85f5448947-c6q9r\" (UID: \"9f4cb38c-f6ba-4250-a710-a465a6d1d74c\") " pod="openshift-console/console-85f5448947-c6q9r" Jul 09 09:41:14.943486 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:14.943397 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9f4cb38c-f6ba-4250-a710-a465a6d1d74c-oauth-serving-cert\") pod \"console-85f5448947-c6q9r\" (UID: \"9f4cb38c-f6ba-4250-a710-a465a6d1d74c\") " pod="openshift-console/console-85f5448947-c6q9r" Jul 09 09:41:14.943486 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:14.943452 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tknkq\" (UniqueName: \"kubernetes.io/projected/9f4cb38c-f6ba-4250-a710-a465a6d1d74c-kube-api-access-tknkq\") pod \"console-85f5448947-c6q9r\" (UID: \"9f4cb38c-f6ba-4250-a710-a465a6d1d74c\") " pod="openshift-console/console-85f5448947-c6q9r" Jul 09 09:41:14.944130 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:14.944103 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9f4cb38c-f6ba-4250-a710-a465a6d1d74c-service-ca\") pod \"console-85f5448947-c6q9r\" (UID: \"9f4cb38c-f6ba-4250-a710-a465a6d1d74c\") " pod="openshift-console/console-85f5448947-c6q9r" Jul 09 09:41:14.944302 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:14.944216 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9f4cb38c-f6ba-4250-a710-a465a6d1d74c-console-config\") pod \"console-85f5448947-c6q9r\" (UID: \"9f4cb38c-f6ba-4250-a710-a465a6d1d74c\") " pod="openshift-console/console-85f5448947-c6q9r" Jul 09 09:41:14.944376 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:14.944194 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9f4cb38c-f6ba-4250-a710-a465a6d1d74c-oauth-serving-cert\") pod \"console-85f5448947-c6q9r\" (UID: \"9f4cb38c-f6ba-4250-a710-a465a6d1d74c\") " pod="openshift-console/console-85f5448947-c6q9r" Jul 09 09:41:14.944414 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:14.944253 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f4cb38c-f6ba-4250-a710-a465a6d1d74c-trusted-ca-bundle\") pod \"console-85f5448947-c6q9r\" (UID: \"9f4cb38c-f6ba-4250-a710-a465a6d1d74c\") " pod="openshift-console/console-85f5448947-c6q9r" Jul 09 09:41:14.945875 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:14.945837 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9f4cb38c-f6ba-4250-a710-a465a6d1d74c-console-oauth-config\") pod \"console-85f5448947-c6q9r\" (UID: \"9f4cb38c-f6ba-4250-a710-a465a6d1d74c\") " pod="openshift-console/console-85f5448947-c6q9r" Jul 09 09:41:14.946064 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:14.946047 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9f4cb38c-f6ba-4250-a710-a465a6d1d74c-console-serving-cert\") pod \"console-85f5448947-c6q9r\" (UID: \"9f4cb38c-f6ba-4250-a710-a465a6d1d74c\") " pod="openshift-console/console-85f5448947-c6q9r" Jul 09 09:41:14.954351 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:14.954333 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tknkq\" (UniqueName: \"kubernetes.io/projected/9f4cb38c-f6ba-4250-a710-a465a6d1d74c-kube-api-access-tknkq\") pod \"console-85f5448947-c6q9r\" (UID: \"9f4cb38c-f6ba-4250-a710-a465a6d1d74c\") " pod="openshift-console/console-85f5448947-c6q9r" Jul 09 09:41:15.059580 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:15.059488 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85f5448947-c6q9r" Jul 09 09:41:15.194927 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:15.194744 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85f5448947-c6q9r"] Jul 09 09:41:15.196785 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:41:15.196760 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f4cb38c_f6ba_4250_a710_a465a6d1d74c.slice/crio-098ed1fe636b8da4a02e99140b7a881de5e11c9dc6ca3095cd684f27008c296b WatchSource:0}: Error finding container 098ed1fe636b8da4a02e99140b7a881de5e11c9dc6ca3095cd684f27008c296b: Status 404 returned error can't find the container with id 098ed1fe636b8da4a02e99140b7a881de5e11c9dc6ca3095cd684f27008c296b Jul 09 09:41:15.296326 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:15.296285 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85f5448947-c6q9r" event={"ID":"9f4cb38c-f6ba-4250-a710-a465a6d1d74c","Type":"ContainerStarted","Data":"c34ea9bdb4fc65b7fc543d3514737d61eda99eea175f3ba5e93cf83fc33e104a"} Jul 09 09:41:15.296473 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:15.296339 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85f5448947-c6q9r" event={"ID":"9f4cb38c-f6ba-4250-a710-a465a6d1d74c","Type":"ContainerStarted","Data":"098ed1fe636b8da4a02e99140b7a881de5e11c9dc6ca3095cd684f27008c296b"} Jul 09 09:41:15.343072 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:15.342978 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-85f5448947-c6q9r" podStartSLOduration=1.342962561 podStartE2EDuration="1.342962561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:41:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:41:15.342404369 +0000 UTC m=+460.434158289" watchObservedRunningTime="2026-07-09 09:41:15.342962561 +0000 UTC m=+460.434716480" Jul 09 09:41:25.060534 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:25.060499 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-85f5448947-c6q9r" Jul 09 09:41:25.061108 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:25.060706 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-85f5448947-c6q9r" Jul 09 09:41:25.065548 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:25.065524 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-85f5448947-c6q9r" Jul 09 09:41:25.336371 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:25.336293 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-85f5448947-c6q9r" Jul 09 09:41:31.654186 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:31.654151 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-fd65d7d6-r9xks"] Jul 09 09:41:31.657105 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:31.657090 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-r9xks" Jul 09 09:41:31.660114 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:31.660094 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-f4mhs\"" Jul 09 09:41:31.661307 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:31.661293 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\"" Jul 09 09:41:31.681658 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:31.681633 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-r9xks"] Jul 09 09:41:31.783332 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:31.783303 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvdcp\" (UniqueName: \"kubernetes.io/projected/65076f9d-75bb-4e52-96c5-36542a147781-kube-api-access-xvdcp\") pod \"model-serving-api-fd65d7d6-r9xks\" (UID: \"65076f9d-75bb-4e52-96c5-36542a147781\") " pod="kserve/model-serving-api-fd65d7d6-r9xks" Jul 09 09:41:31.783495 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:31.783370 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65076f9d-75bb-4e52-96c5-36542a147781-tls-certs\") pod \"model-serving-api-fd65d7d6-r9xks\" (UID: \"65076f9d-75bb-4e52-96c5-36542a147781\") " pod="kserve/model-serving-api-fd65d7d6-r9xks" Jul 09 09:41:31.884158 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:31.884129 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvdcp\" (UniqueName: \"kubernetes.io/projected/65076f9d-75bb-4e52-96c5-36542a147781-kube-api-access-xvdcp\") pod \"model-serving-api-fd65d7d6-r9xks\" (UID: \"65076f9d-75bb-4e52-96c5-36542a147781\") " pod="kserve/model-serving-api-fd65d7d6-r9xks" Jul 09 09:41:31.884287 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:31.884177 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65076f9d-75bb-4e52-96c5-36542a147781-tls-certs\") pod \"model-serving-api-fd65d7d6-r9xks\" (UID: \"65076f9d-75bb-4e52-96c5-36542a147781\") " pod="kserve/model-serving-api-fd65d7d6-r9xks" Jul 09 09:41:31.886725 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:31.886706 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65076f9d-75bb-4e52-96c5-36542a147781-tls-certs\") pod \"model-serving-api-fd65d7d6-r9xks\" (UID: \"65076f9d-75bb-4e52-96c5-36542a147781\") " pod="kserve/model-serving-api-fd65d7d6-r9xks" Jul 09 09:41:31.896793 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:31.896774 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvdcp\" (UniqueName: \"kubernetes.io/projected/65076f9d-75bb-4e52-96c5-36542a147781-kube-api-access-xvdcp\") pod \"model-serving-api-fd65d7d6-r9xks\" (UID: \"65076f9d-75bb-4e52-96c5-36542a147781\") " pod="kserve/model-serving-api-fd65d7d6-r9xks" Jul 09 09:41:31.967375 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:31.967317 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-r9xks" Jul 09 09:41:32.098986 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:32.098956 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-r9xks"] Jul 09 09:41:32.101705 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:41:32.101677 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65076f9d_75bb_4e52_96c5_36542a147781.slice/crio-c9bba1ca5879fa7cc5c006d7da5b9062a56a788c2b29218f5cfa74c9666c67a3 WatchSource:0}: Error finding container c9bba1ca5879fa7cc5c006d7da5b9062a56a788c2b29218f5cfa74c9666c67a3: Status 404 returned error can't find the container with id c9bba1ca5879fa7cc5c006d7da5b9062a56a788c2b29218f5cfa74c9666c67a3 Jul 09 09:41:32.360177 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:32.360088 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-r9xks" event={"ID":"65076f9d-75bb-4e52-96c5-36542a147781","Type":"ContainerStarted","Data":"c9bba1ca5879fa7cc5c006d7da5b9062a56a788c2b29218f5cfa74c9666c67a3"} Jul 09 09:41:37.381603 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:37.381565 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-r9xks" event={"ID":"65076f9d-75bb-4e52-96c5-36542a147781","Type":"ContainerStarted","Data":"c7817e48bf033420acdd478d123714b6d61500114a5de81e0475e4a20d35c2d6"} Jul 09 09:41:37.382030 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:37.381713 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-fd65d7d6-r9xks" Jul 09 09:41:37.402402 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:37.402353 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-fd65d7d6-r9xks" podStartSLOduration=1.620105774 podStartE2EDuration="6.402340031s" totalImagesPullingTime="4.782234257s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:41:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:41:37.401973072 +0000 UTC m=+482.493726991" watchObservedRunningTime="2026-07-09 09:41:37.402340031 +0000 UTC m=+482.494093951" Jul 09 09:41:47.487169 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:47.487136 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-rs6b4"] Jul 09 09:41:47.490505 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:47.490488 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-rs6b4" Jul 09 09:41:47.499877 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:47.499834 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-rs6b4"] Jul 09 09:41:47.611678 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:47.611639 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0584ca51-2a1c-427c-a258-6947af590a68-model-cache\") pod \"s3-init-rs6b4\" (UID: \"0584ca51-2a1c-427c-a258-6947af590a68\") " pod="kserve/s3-init-rs6b4" Jul 09 09:41:47.611883 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:47.611792 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jcbv\" (UniqueName: \"kubernetes.io/projected/0584ca51-2a1c-427c-a258-6947af590a68-kube-api-access-2jcbv\") pod \"s3-init-rs6b4\" (UID: \"0584ca51-2a1c-427c-a258-6947af590a68\") " pod="kserve/s3-init-rs6b4" Jul 09 09:41:47.712257 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:47.712223 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jcbv\" (UniqueName: \"kubernetes.io/projected/0584ca51-2a1c-427c-a258-6947af590a68-kube-api-access-2jcbv\") pod \"s3-init-rs6b4\" (UID: \"0584ca51-2a1c-427c-a258-6947af590a68\") " pod="kserve/s3-init-rs6b4" Jul 09 09:41:47.712436 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:47.712292 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0584ca51-2a1c-427c-a258-6947af590a68-model-cache\") pod \"s3-init-rs6b4\" (UID: \"0584ca51-2a1c-427c-a258-6947af590a68\") " pod="kserve/s3-init-rs6b4" Jul 09 09:41:47.712656 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:47.712636 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0584ca51-2a1c-427c-a258-6947af590a68-model-cache\") pod \"s3-init-rs6b4\" (UID: \"0584ca51-2a1c-427c-a258-6947af590a68\") " pod="kserve/s3-init-rs6b4" Jul 09 09:41:47.721236 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:47.721206 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jcbv\" (UniqueName: \"kubernetes.io/projected/0584ca51-2a1c-427c-a258-6947af590a68-kube-api-access-2jcbv\") pod \"s3-init-rs6b4\" (UID: \"0584ca51-2a1c-427c-a258-6947af590a68\") " pod="kserve/s3-init-rs6b4" Jul 09 09:41:47.816318 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:47.816259 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-rs6b4" Jul 09 09:41:47.944190 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:47.944168 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-rs6b4"] Jul 09 09:41:47.946314 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:41:47.946283 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0584ca51_2a1c_427c_a258_6947af590a68.slice/crio-30b58cf116150301ca215a42f8abb60d4ef76c78c1d771b4962998ef4a1ab483 WatchSource:0}: Error finding container 30b58cf116150301ca215a42f8abb60d4ef76c78c1d771b4962998ef4a1ab483: Status 404 returned error can't find the container with id 30b58cf116150301ca215a42f8abb60d4ef76c78c1d771b4962998ef4a1ab483 Jul 09 09:41:48.389800 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:48.389768 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-fd65d7d6-r9xks" Jul 09 09:41:48.428671 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:48.428639 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-rs6b4" event={"ID":"0584ca51-2a1c-427c-a258-6947af590a68","Type":"ContainerStarted","Data":"30b58cf116150301ca215a42f8abb60d4ef76c78c1d771b4962998ef4a1ab483"} Jul 09 09:41:52.450322 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:52.450285 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-rs6b4" event={"ID":"0584ca51-2a1c-427c-a258-6947af590a68","Type":"ContainerStarted","Data":"45440b7fe35306d9cfc412491d5a1e3fedf06fc0cdd2cc21d0a806f0b34b6092"} Jul 09 09:41:53.454911 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:53.454881 2578 generic.go:356] "Generic (PLEG): container finished" podID="0584ca51-2a1c-427c-a258-6947af590a68" containerID="45440b7fe35306d9cfc412491d5a1e3fedf06fc0cdd2cc21d0a806f0b34b6092" exitCode=0 Jul 09 09:41:53.455284 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:53.454954 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-rs6b4" event={"ID":"0584ca51-2a1c-427c-a258-6947af590a68","Type":"ContainerDied","Data":"45440b7fe35306d9cfc412491d5a1e3fedf06fc0cdd2cc21d0a806f0b34b6092"} Jul 09 09:41:58.477983 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:58.477904 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-rs6b4" event={"ID":"0584ca51-2a1c-427c-a258-6947af590a68","Type":"ContainerStarted","Data":"15ef0f90cd5247be7e50a07f34c5d342ec8fa57e8bb82f42aff4533525fea834"} Jul 09 09:41:58.495599 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:41:58.495540 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-rs6b4" podStartSLOduration=1.80346975 podStartE2EDuration="11.495523036s" totalImagesPullingTime="8.301674179s" totalInitContainerRuntime="1.390379107s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:41:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:41:58.494255592 +0000 UTC m=+503.586009515" watchObservedRunningTime="2026-07-09 09:41:58.495523036 +0000 UTC m=+503.587276959" Jul 09 09:43:03.720615 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:03.720579 2578 generic.go:356] "Generic (PLEG): container finished" podID="0584ca51-2a1c-427c-a258-6947af590a68" containerID="15ef0f90cd5247be7e50a07f34c5d342ec8fa57e8bb82f42aff4533525fea834" exitCode=0 Jul 09 09:43:03.721043 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:03.720646 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-rs6b4" event={"ID":"0584ca51-2a1c-427c-a258-6947af590a68","Type":"ContainerDied","Data":"15ef0f90cd5247be7e50a07f34c5d342ec8fa57e8bb82f42aff4533525fea834"} Jul 09 09:43:04.849813 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:04.849793 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-rs6b4" Jul 09 09:43:04.926609 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:04.926525 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0584ca51-2a1c-427c-a258-6947af590a68-model-cache\") pod \"0584ca51-2a1c-427c-a258-6947af590a68\" (UID: \"0584ca51-2a1c-427c-a258-6947af590a68\") " Jul 09 09:43:04.926737 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:04.926644 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2jcbv\" (UniqueName: \"kubernetes.io/projected/0584ca51-2a1c-427c-a258-6947af590a68-kube-api-access-2jcbv\") pod \"0584ca51-2a1c-427c-a258-6947af590a68\" (UID: \"0584ca51-2a1c-427c-a258-6947af590a68\") " Jul 09 09:43:04.929039 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:04.929010 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0584ca51-2a1c-427c-a258-6947af590a68-kube-api-access-2jcbv" (OuterVolumeSpecName: "kube-api-access-2jcbv") pod "0584ca51-2a1c-427c-a258-6947af590a68" (UID: "0584ca51-2a1c-427c-a258-6947af590a68"). InnerVolumeSpecName "kube-api-access-2jcbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 09:43:04.944968 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:04.944949 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0584ca51-2a1c-427c-a258-6947af590a68-model-cache" (OuterVolumeSpecName: "model-cache") pod "0584ca51-2a1c-427c-a258-6947af590a68" (UID: "0584ca51-2a1c-427c-a258-6947af590a68"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 09:43:05.027256 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:05.027198 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0584ca51-2a1c-427c-a258-6947af590a68-model-cache\") on node \"ip-10-0-135-204.ec2.internal\" DevicePath \"\"" Jul 09 09:43:05.027256 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:05.027218 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jcbv\" (UniqueName: \"kubernetes.io/projected/0584ca51-2a1c-427c-a258-6947af590a68-kube-api-access-2jcbv\") on node \"ip-10-0-135-204.ec2.internal\" DevicePath \"\"" Jul 09 09:43:05.729733 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:05.729653 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-rs6b4" Jul 09 09:43:05.729733 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:05.729662 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-rs6b4" event={"ID":"0584ca51-2a1c-427c-a258-6947af590a68","Type":"ContainerDied","Data":"30b58cf116150301ca215a42f8abb60d4ef76c78c1d771b4962998ef4a1ab483"} Jul 09 09:43:05.729733 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:05.729698 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30b58cf116150301ca215a42f8abb60d4ef76c78c1d771b4962998ef4a1ab483" Jul 09 09:43:18.359147 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:18.359117 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7c964dbd44-rn899"] Jul 09 09:43:18.359501 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:18.359403 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0584ca51-2a1c-427c-a258-6947af590a68" containerName="s3-init" Jul 09 09:43:18.359501 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:18.359413 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0584ca51-2a1c-427c-a258-6947af590a68" containerName="s3-init" Jul 09 09:43:18.359501 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:18.359434 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0584ca51-2a1c-427c-a258-6947af590a68" containerName="download-hf-model" Jul 09 09:43:18.359501 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:18.359440 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0584ca51-2a1c-427c-a258-6947af590a68" containerName="download-hf-model" Jul 09 09:43:18.359501 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:18.359491 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0584ca51-2a1c-427c-a258-6947af590a68" containerName="s3-init" Jul 09 09:43:18.362759 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:18.362738 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7c964dbd44-rn899" Jul 09 09:43:18.365420 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:18.365382 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 09 09:43:18.365542 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:18.365435 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Jul 09 09:43:18.365542 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:18.365501 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Jul 09 09:43:18.365542 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:18.365505 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-predictor-serving-cert\"" Jul 09 09:43:18.365542 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:18.365511 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\"" Jul 09 09:43:18.366299 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:18.366284 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tmt4d\"" Jul 09 09:43:18.366392 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:18.366297 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Jul 09 09:43:18.375953 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:18.375935 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7c964dbd44-rn899"] Jul 09 09:43:18.435568 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:18.435525 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69g6k\" (UniqueName: \"kubernetes.io/projected/bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc-kube-api-access-69g6k\") pod \"isvc-sklearn-graph-1-predictor-7c964dbd44-rn899\" (UID: \"bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7c964dbd44-rn899" Jul 09 09:43:18.435709 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:18.435587 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-7c964dbd44-rn899\" (UID: \"bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7c964dbd44-rn899" Jul 09 09:43:18.435709 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:18.435618 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-7c964dbd44-rn899\" (UID: \"bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7c964dbd44-rn899" Jul 09 09:43:18.435709 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:18.435683 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-7c964dbd44-rn899\" (UID: \"bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7c964dbd44-rn899" Jul 09 09:43:18.435813 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:18.435722 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-7c964dbd44-rn899\" (UID: \"bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7c964dbd44-rn899" Jul 09 09:43:18.536909 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:18.536877 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-7c964dbd44-rn899\" (UID: \"bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7c964dbd44-rn899" Jul 09 09:43:18.537092 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:18.536954 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-7c964dbd44-rn899\" (UID: \"bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7c964dbd44-rn899" Jul 09 09:43:18.537158 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:18.537093 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-7c964dbd44-rn899\" (UID: \"bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7c964dbd44-rn899" Jul 09 09:43:18.537216 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:18.537155 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-7c964dbd44-rn899\" (UID: \"bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7c964dbd44-rn899" Jul 09 09:43:18.537271 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:43:18.537242 2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-serving-cert: secret "isvc-sklearn-graph-1-predictor-serving-cert" not found Jul 09 09:43:18.537330 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:18.537268 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69g6k\" (UniqueName: \"kubernetes.io/projected/bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc-kube-api-access-69g6k\") pod \"isvc-sklearn-graph-1-predictor-7c964dbd44-rn899\" (UID: \"bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7c964dbd44-rn899" Jul 09 09:43:18.537330 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:43:18.537297 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc-proxy-tls podName:bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc nodeName:}" failed. No retries permitted until 2026-07-09 09:43:19.037279314 +0000 UTC m=+584.129033213 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc-proxy-tls") pod "isvc-sklearn-graph-1-predictor-7c964dbd44-rn899" (UID: "bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc") : secret "isvc-sklearn-graph-1-predictor-serving-cert" not found Jul 09 09:43:18.537816 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:18.537596 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-7c964dbd44-rn899\" (UID: \"bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7c964dbd44-rn899" Jul 09 09:43:18.537816 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:18.537770 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-7c964dbd44-rn899\" (UID: \"bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7c964dbd44-rn899" Jul 09 09:43:18.538009 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:18.537920 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-7c964dbd44-rn899\" (UID: \"bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7c964dbd44-rn899" Jul 09 09:43:18.545950 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:18.545928 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69g6k\" (UniqueName: \"kubernetes.io/projected/bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc-kube-api-access-69g6k\") pod \"isvc-sklearn-graph-1-predictor-7c964dbd44-rn899\" (UID: \"bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7c964dbd44-rn899" Jul 09 09:43:18.669000 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:18.668969 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-4cp66"] Jul 09 09:43:18.672396 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:18.672381 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-4cp66" Jul 09 09:43:18.674809 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:18.674775 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-predictor-serving-cert\"" Jul 09 09:43:18.674985 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:18.674967 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-kube-rbac-proxy-sar-config\"" Jul 09 09:43:18.683238 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:18.683214 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-4cp66"] Jul 09 09:43:18.739859 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:18.739809 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq8gx\" (UniqueName: \"kubernetes.io/projected/953cb958-4294-4c91-912a-3d4ac0bf4c67-kube-api-access-sq8gx\") pod \"isvc-xgboost-graph-predictor-555fc69d56-4cp66\" (UID: \"953cb958-4294-4c91-912a-3d4ac0bf4c67\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-4cp66" Jul 09 09:43:18.740012 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:18.739869 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/953cb958-4294-4c91-912a-3d4ac0bf4c67-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-4cp66\" (UID: \"953cb958-4294-4c91-912a-3d4ac0bf4c67\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-4cp66" Jul 09 09:43:18.740012 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:18.739917 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/953cb958-4294-4c91-912a-3d4ac0bf4c67-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-4cp66\" (UID: \"953cb958-4294-4c91-912a-3d4ac0bf4c67\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-4cp66" Jul 09 09:43:18.740012 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:18.739973 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/953cb958-4294-4c91-912a-3d4ac0bf4c67-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-4cp66\" (UID: \"953cb958-4294-4c91-912a-3d4ac0bf4c67\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-4cp66" Jul 09 09:43:18.740012 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:18.739992 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/953cb958-4294-4c91-912a-3d4ac0bf4c67-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-4cp66\" (UID: \"953cb958-4294-4c91-912a-3d4ac0bf4c67\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-4cp66" Jul 09 09:43:18.769995 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:18.769967 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-84b568d556-46fps"] Jul 09 09:43:18.773362 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:18.773347 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-84b568d556-46fps" Jul 09 09:43:18.775756 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:18.775736 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-predictor-serving-cert\"" Jul 09 09:43:18.775890 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:18.775795 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\"" Jul 09 09:43:18.783908 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:18.783886 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-84b568d556-46fps"] Jul 09 09:43:18.840649 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:18.840626 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-84b568d556-46fps\" (UID: \"c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-84b568d556-46fps" Jul 09 09:43:18.840769 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:18.840664 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-84b568d556-46fps\" (UID: \"c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-84b568d556-46fps" Jul 09 09:43:18.840769 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:18.840685 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-84b568d556-46fps\" (UID: \"c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-84b568d556-46fps" Jul 09 09:43:18.840872 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:18.840767 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6scc\" (UniqueName: \"kubernetes.io/projected/c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca-kube-api-access-j6scc\") pod \"isvc-sklearn-graph-2-predictor-84b568d556-46fps\" (UID: \"c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-84b568d556-46fps" Jul 09 09:43:18.840872 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:18.840816 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq8gx\" (UniqueName: \"kubernetes.io/projected/953cb958-4294-4c91-912a-3d4ac0bf4c67-kube-api-access-sq8gx\") pod \"isvc-xgboost-graph-predictor-555fc69d56-4cp66\" (UID: \"953cb958-4294-4c91-912a-3d4ac0bf4c67\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-4cp66" Jul 09 09:43:18.840872 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:18.840866 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/953cb958-4294-4c91-912a-3d4ac0bf4c67-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-4cp66\" (UID: \"953cb958-4294-4c91-912a-3d4ac0bf4c67\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-4cp66" Jul 09 09:43:18.840980 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:18.840890 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/953cb958-4294-4c91-912a-3d4ac0bf4c67-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-4cp66\" (UID: \"953cb958-4294-4c91-912a-3d4ac0bf4c67\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-4cp66" Jul 09 09:43:18.840980 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:18.840920 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-84b568d556-46fps\" (UID: \"c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-84b568d556-46fps" Jul 09 09:43:18.840980 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:18.840939 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/953cb958-4294-4c91-912a-3d4ac0bf4c67-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-4cp66\" (UID: \"953cb958-4294-4c91-912a-3d4ac0bf4c67\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-4cp66" Jul 09 09:43:18.840980 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:18.840955 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/953cb958-4294-4c91-912a-3d4ac0bf4c67-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-4cp66\" (UID: \"953cb958-4294-4c91-912a-3d4ac0bf4c67\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-4cp66" Jul 09 09:43:18.841164 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:43:18.841027 2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-predictor-serving-cert: secret "isvc-xgboost-graph-predictor-serving-cert" not found Jul 09 09:43:18.841164 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:43:18.841103 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/953cb958-4294-4c91-912a-3d4ac0bf4c67-proxy-tls podName:953cb958-4294-4c91-912a-3d4ac0bf4c67 nodeName:}" failed. No retries permitted until 2026-07-09 09:43:19.341064954 +0000 UTC m=+584.432818855 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/953cb958-4294-4c91-912a-3d4ac0bf4c67-proxy-tls") pod "isvc-xgboost-graph-predictor-555fc69d56-4cp66" (UID: "953cb958-4294-4c91-912a-3d4ac0bf4c67") : secret "isvc-xgboost-graph-predictor-serving-cert" not found Jul 09 09:43:18.841316 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:18.841296 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/953cb958-4294-4c91-912a-3d4ac0bf4c67-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-4cp66\" (UID: \"953cb958-4294-4c91-912a-3d4ac0bf4c67\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-4cp66" Jul 09 09:43:18.841625 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:18.841604 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/953cb958-4294-4c91-912a-3d4ac0bf4c67-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-4cp66\" (UID: \"953cb958-4294-4c91-912a-3d4ac0bf4c67\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-4cp66" Jul 09 09:43:18.841693 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:18.841607 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/953cb958-4294-4c91-912a-3d4ac0bf4c67-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-4cp66\" (UID: \"953cb958-4294-4c91-912a-3d4ac0bf4c67\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-4cp66" Jul 09 09:43:18.851085 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:18.851059 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq8gx\" (UniqueName: \"kubernetes.io/projected/953cb958-4294-4c91-912a-3d4ac0bf4c67-kube-api-access-sq8gx\") pod \"isvc-xgboost-graph-predictor-555fc69d56-4cp66\" (UID: \"953cb958-4294-4c91-912a-3d4ac0bf4c67\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-4cp66" Jul 09 09:43:18.942363 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:18.942300 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-84b568d556-46fps\" (UID: \"c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-84b568d556-46fps" Jul 09 09:43:18.942363 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:18.942337 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-84b568d556-46fps\" (UID: \"c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-84b568d556-46fps" Jul 09 09:43:18.942511 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:18.942366 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-84b568d556-46fps\" (UID: \"c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-84b568d556-46fps" Jul 09 09:43:18.942511 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:18.942418 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6scc\" (UniqueName: \"kubernetes.io/projected/c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca-kube-api-access-j6scc\") pod \"isvc-sklearn-graph-2-predictor-84b568d556-46fps\" (UID: \"c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-84b568d556-46fps" Jul 09 09:43:18.942590 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:18.942513 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-84b568d556-46fps\" (UID: \"c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-84b568d556-46fps" Jul 09 09:43:18.942881 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:18.942831 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-84b568d556-46fps\" (UID: \"c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-84b568d556-46fps" Jul 09 09:43:18.942962 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:18.942950 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-84b568d556-46fps\" (UID: \"c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-84b568d556-46fps" Jul 09 09:43:18.943005 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:18.942993 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-84b568d556-46fps\" (UID: \"c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-84b568d556-46fps" Jul 09 09:43:18.944882 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:18.944863 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-84b568d556-46fps\" (UID: \"c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-84b568d556-46fps" Jul 09 09:43:18.949879 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:18.949859 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6scc\" (UniqueName: \"kubernetes.io/projected/c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca-kube-api-access-j6scc\") pod \"isvc-sklearn-graph-2-predictor-84b568d556-46fps\" (UID: \"c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-84b568d556-46fps" Jul 09 09:43:19.043195 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:19.043163 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-7c964dbd44-rn899\" (UID: \"bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7c964dbd44-rn899" Jul 09 09:43:19.045562 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:19.045532 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-7c964dbd44-rn899\" (UID: \"bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7c964dbd44-rn899" Jul 09 09:43:19.083584 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:19.083561 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-84b568d556-46fps" Jul 09 09:43:19.221492 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:43:19.221439 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8d2c03e_cf8b_4bcc_b354_11e93e4b3bca.slice/crio-f9a2ed1c63b420afb434b6025b4e6d199e86bc2b2354ac74d01736a1d3cdbbaf WatchSource:0}: Error finding container f9a2ed1c63b420afb434b6025b4e6d199e86bc2b2354ac74d01736a1d3cdbbaf: Status 404 returned error can't find the container with id f9a2ed1c63b420afb434b6025b4e6d199e86bc2b2354ac74d01736a1d3cdbbaf Jul 09 09:43:19.226086 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:19.226067 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-84b568d556-46fps"] Jul 09 09:43:19.272685 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:19.272663 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7c964dbd44-rn899" Jul 09 09:43:19.345928 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:19.345895 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/953cb958-4294-4c91-912a-3d4ac0bf4c67-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-4cp66\" (UID: \"953cb958-4294-4c91-912a-3d4ac0bf4c67\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-4cp66" Jul 09 09:43:19.348303 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:19.348285 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/953cb958-4294-4c91-912a-3d4ac0bf4c67-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-4cp66\" (UID: \"953cb958-4294-4c91-912a-3d4ac0bf4c67\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-4cp66" Jul 09 09:43:19.403517 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:19.403494 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7c964dbd44-rn899"] Jul 09 09:43:19.405736 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:43:19.405713 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd442d1d_ea1c_47cd_9b75_7f6eb7f61cbc.slice/crio-b4adb1525adce6934a5a3251031e0b99043cd9173b92a8b4e8f8675b2eca4561 WatchSource:0}: Error finding container b4adb1525adce6934a5a3251031e0b99043cd9173b92a8b4e8f8675b2eca4561: Status 404 returned error can't find the container with id b4adb1525adce6934a5a3251031e0b99043cd9173b92a8b4e8f8675b2eca4561 Jul 09 09:43:19.582565 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:19.582473 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-4cp66" Jul 09 09:43:19.717833 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:19.717810 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-4cp66"] Jul 09 09:43:19.721240 ip-10-0-135-204 kubenswrapper[2578]: W0709 09:43:19.721214 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod953cb958_4294_4c91_912a_3d4ac0bf4c67.slice/crio-1c2a6934f750f923987608a79ec39f8471827ccc3e1b1e4d881c6e41c13277ad WatchSource:0}: Error finding container 1c2a6934f750f923987608a79ec39f8471827ccc3e1b1e4d881c6e41c13277ad: Status 404 returned error can't find the container with id 1c2a6934f750f923987608a79ec39f8471827ccc3e1b1e4d881c6e41c13277ad Jul 09 09:43:19.782796 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:19.782760 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-4cp66" event={"ID":"953cb958-4294-4c91-912a-3d4ac0bf4c67","Type":"ContainerStarted","Data":"1c2a6934f750f923987608a79ec39f8471827ccc3e1b1e4d881c6e41c13277ad"} Jul 09 09:43:19.784767 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:19.784552 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7c964dbd44-rn899" event={"ID":"bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc","Type":"ContainerStarted","Data":"10383cd652304340bdf01c8a577118c47eb953458ee645770927592f7fe10630"} Jul 09 09:43:19.784767 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:19.784617 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7c964dbd44-rn899" event={"ID":"bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc","Type":"ContainerStarted","Data":"b4adb1525adce6934a5a3251031e0b99043cd9173b92a8b4e8f8675b2eca4561"} Jul 09 09:43:19.786328 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:19.786302 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-84b568d556-46fps" event={"ID":"c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca","Type":"ContainerStarted","Data":"84f18c9e86e6ac3fe35d868f21cc9844d744b12150620f1f0184f8c43a1fa1a5"} Jul 09 09:43:19.786434 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:19.786359 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-84b568d556-46fps" event={"ID":"c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca","Type":"ContainerStarted","Data":"f9a2ed1c63b420afb434b6025b4e6d199e86bc2b2354ac74d01736a1d3cdbbaf"} Jul 09 09:43:20.791532 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:20.791489 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-4cp66" event={"ID":"953cb958-4294-4c91-912a-3d4ac0bf4c67","Type":"ContainerStarted","Data":"01132d0ee444236160f5e6d4681915ada5c540f9f7887bb31e3dfda4515de0a1"} Jul 09 09:43:23.805475 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:23.805433 2578 generic.go:356] "Generic (PLEG): container finished" podID="953cb958-4294-4c91-912a-3d4ac0bf4c67" containerID="01132d0ee444236160f5e6d4681915ada5c540f9f7887bb31e3dfda4515de0a1" exitCode=0 Jul 09 09:43:23.805832 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:23.805505 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-4cp66" event={"ID":"953cb958-4294-4c91-912a-3d4ac0bf4c67","Type":"ContainerDied","Data":"01132d0ee444236160f5e6d4681915ada5c540f9f7887bb31e3dfda4515de0a1"} Jul 09 09:43:23.807145 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:23.807121 2578 generic.go:356] "Generic (PLEG): container finished" podID="bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc" containerID="10383cd652304340bdf01c8a577118c47eb953458ee645770927592f7fe10630" exitCode=0 Jul 09 09:43:23.807240 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:23.807181 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7c964dbd44-rn899" event={"ID":"bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc","Type":"ContainerDied","Data":"10383cd652304340bdf01c8a577118c47eb953458ee645770927592f7fe10630"} Jul 09 09:43:23.808904 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:23.808888 2578 generic.go:356] "Generic (PLEG): container finished" podID="c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca" containerID="84f18c9e86e6ac3fe35d868f21cc9844d744b12150620f1f0184f8c43a1fa1a5" exitCode=0 Jul 09 09:43:23.808989 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:23.808922 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-84b568d556-46fps" event={"ID":"c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca","Type":"ContainerDied","Data":"84f18c9e86e6ac3fe35d868f21cc9844d744b12150620f1f0184f8c43a1fa1a5"} Jul 09 09:43:51.688788 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:51.688761 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xf4sw_1677420d-7376-45ad-ac67-20128aec5756/ovn-acl-logging/0.log" Jul 09 09:43:51.689234 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:51.688923 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xf4sw_1677420d-7376-45ad-ac67-20128aec5756/ovn-acl-logging/0.log" Jul 09 09:43:51.989577 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:51.989488 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7c964dbd44-rn899" event={"ID":"bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc","Type":"ContainerStarted","Data":"1740f57cbe1cb96fba0c509be76bc8b8bbe754bcbe543aa4c8230aeb5b4e071d"} Jul 09 09:43:51.991200 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:51.991180 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-84b568d556-46fps" event={"ID":"c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca","Type":"ContainerStarted","Data":"0e1b7c13bb308a3c00c2e2dcf157ed9f9f82ca8a33d3a33945fef17058d1d3ec"} Jul 09 09:43:53.005931 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:53.005895 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-4cp66" event={"ID":"953cb958-4294-4c91-912a-3d4ac0bf4c67","Type":"ContainerStarted","Data":"699676e75c2a0c2b92b1d769beb5633e5ba576fb802cd252f8db661feee3ee7b"} Jul 09 09:43:55.016110 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:55.016069 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-84b568d556-46fps" event={"ID":"c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca","Type":"ContainerStarted","Data":"ccbe677866490ccb27824b096764b8585344631c045eb531c3952b03ad27a7bb"} Jul 09 09:43:55.016522 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:55.016246 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-84b568d556-46fps" Jul 09 09:43:55.018171 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:55.018145 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-4cp66" event={"ID":"953cb958-4294-4c91-912a-3d4ac0bf4c67","Type":"ContainerStarted","Data":"d3039f4e9adabdc9905284da36df318dd2bdfad8d441f28c9b0981c252093442"} Jul 09 09:43:55.018302 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:55.018272 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-4cp66" Jul 09 09:43:55.018381 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:55.018305 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-4cp66" Jul 09 09:43:55.020128 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:55.020097 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-4cp66" podUID="953cb958-4294-4c91-912a-3d4ac0bf4c67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused" Jul 09 09:43:55.020293 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:55.020273 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7c964dbd44-rn899" event={"ID":"bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc","Type":"ContainerStarted","Data":"30c4a9b7c5593cdc5225e3df859b15a2e55c66160864e5f573387063efa1862b"} Jul 09 09:43:55.020722 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:55.020516 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7c964dbd44-rn899" Jul 09 09:43:55.020722 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:55.020553 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7c964dbd44-rn899" Jul 09 09:43:55.021636 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:55.021616 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7c964dbd44-rn899" podUID="bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused" Jul 09 09:43:55.050886 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:55.050759 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-84b568d556-46fps" podStartSLOduration=2.5954985329999998 podStartE2EDuration="37.050742847s" totalImagesPullingTime="30.878498096s" totalInitContainerRuntime="3.576746218s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:43:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:43:55.049472711 +0000 UTC m=+620.141226660" watchObservedRunningTime="2026-07-09 09:43:55.050742847 +0000 UTC m=+620.142496767" Jul 09 09:43:55.086670 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:55.086625 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-4cp66" podStartSLOduration=2.281179672 podStartE2EDuration="37.086611525s" totalImagesPullingTime="30.870693185s" totalInitContainerRuntime="3.934738668s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:43:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:43:55.086114744 +0000 UTC m=+620.177868663" watchObservedRunningTime="2026-07-09 09:43:55.086611525 +0000 UTC m=+620.178365444" Jul 09 09:43:55.119517 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:55.119465 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7c964dbd44-rn899" podStartSLOduration=2.56033416 podStartE2EDuration="37.11944978s" totalImagesPullingTime="30.879665742s" totalInitContainerRuntime="3.679449878s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:43:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:43:55.118597815 +0000 UTC m=+620.210351746" watchObservedRunningTime="2026-07-09 09:43:55.11944978 +0000 UTC m=+620.211203701" Jul 09 09:43:56.024378 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:56.024331 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7c964dbd44-rn899" podUID="bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused" Jul 09 09:43:56.024821 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:56.024464 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-4cp66" podUID="953cb958-4294-4c91-912a-3d4ac0bf4c67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused" Jul 09 09:43:56.024821 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:56.024712 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-84b568d556-46fps" Jul 09 09:43:56.025458 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:56.025436 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-84b568d556-46fps" podUID="c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused" Jul 09 09:43:57.027786 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:43:57.027743 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-84b568d556-46fps" podUID="c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused" Jul 09 09:44:01.028738 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:44:01.028711 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7c964dbd44-rn899" Jul 09 09:44:01.029179 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:44:01.029053 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-4cp66" Jul 09 09:44:01.029267 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:44:01.029175 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7c964dbd44-rn899" podUID="bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused" Jul 09 09:44:01.029483 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:44:01.029459 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-4cp66" podUID="953cb958-4294-4c91-912a-3d4ac0bf4c67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused" Jul 09 09:44:02.031918 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:44:02.031890 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-84b568d556-46fps" Jul 09 09:44:02.032462 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:44:02.032436 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-84b568d556-46fps" podUID="c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused" Jul 09 09:44:11.029789 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:44:11.029748 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-4cp66" podUID="953cb958-4294-4c91-912a-3d4ac0bf4c67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused" Jul 09 09:44:11.030260 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:44:11.029755 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7c964dbd44-rn899" podUID="bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused" Jul 09 09:44:12.032885 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:44:12.032830 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-84b568d556-46fps" podUID="c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused" Jul 09 09:44:21.029231 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:44:21.029192 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7c964dbd44-rn899" podUID="bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused" Jul 09 09:44:21.516225 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:44:21.029448 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-4cp66" podUID="953cb958-4294-4c91-912a-3d4ac0bf4c67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused" Jul 09 09:44:22.032980 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:44:22.032937 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-84b568d556-46fps" podUID="c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused" Jul 09 09:44:31.029272 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:44:31.029230 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7c964dbd44-rn899" podUID="bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused" Jul 09 09:44:31.029701 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:44:31.029462 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-4cp66" podUID="953cb958-4294-4c91-912a-3d4ac0bf4c67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused" Jul 09 09:44:32.032983 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:44:32.032946 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-84b568d556-46fps" podUID="c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused" Jul 09 09:44:41.029469 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:44:41.029432 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7c964dbd44-rn899" podUID="bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused" Jul 09 09:44:41.029818 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:44:41.029435 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-4cp66" podUID="953cb958-4294-4c91-912a-3d4ac0bf4c67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused" Jul 09 09:44:42.032521 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:44:42.032486 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-84b568d556-46fps" podUID="c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused" Jul 09 09:44:51.029799 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:44:51.029761 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7c964dbd44-rn899" podUID="bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused" Jul 09 09:44:51.030652 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:44:51.030632 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-4cp66" Jul 09 09:44:52.032525 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:44:52.032476 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-84b568d556-46fps" podUID="c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused" Jul 09 09:45:01.029997 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:01.029955 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7c964dbd44-rn899" podUID="bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused" Jul 09 09:45:02.032595 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:02.032549 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-84b568d556-46fps" podUID="c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused" Jul 09 09:45:11.031007 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:11.030980 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7c964dbd44-rn899" Jul 09 09:45:12.033998 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:12.033970 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-84b568d556-46fps" Jul 09 09:45:48.401792 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:48.401765 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-4cp66"] Jul 09 09:45:48.402366 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:48.402087 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-4cp66" podUID="953cb958-4294-4c91-912a-3d4ac0bf4c67" containerName="kserve-container" containerID="cri-o://699676e75c2a0c2b92b1d769beb5633e5ba576fb802cd252f8db661feee3ee7b" gracePeriod=30 Jul 09 09:45:48.402366 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:48.402165 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-4cp66" podUID="953cb958-4294-4c91-912a-3d4ac0bf4c67" containerName="kube-rbac-proxy" containerID="cri-o://d3039f4e9adabdc9905284da36df318dd2bdfad8d441f28c9b0981c252093442" gracePeriod=30 Jul 09 09:45:48.445065 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:48.445039 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-84b568d556-46fps"] Jul 09 09:45:48.445347 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:48.445315 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-84b568d556-46fps" podUID="c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca" containerName="kserve-container" containerID="cri-o://0e1b7c13bb308a3c00c2e2dcf157ed9f9f82ca8a33d3a33945fef17058d1d3ec" gracePeriod=30 Jul 09 09:45:48.445458 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:48.445339 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-84b568d556-46fps" podUID="c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca" containerName="kube-rbac-proxy" containerID="cri-o://ccbe677866490ccb27824b096764b8585344631c045eb531c3952b03ad27a7bb" gracePeriod=30 Jul 09 09:45:48.498822 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:48.498781 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7c964dbd44-rn899"] Jul 09 09:45:48.499037 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:48.499015 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7c964dbd44-rn899" podUID="bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc" containerName="kserve-container" containerID="cri-o://1740f57cbe1cb96fba0c509be76bc8b8bbe754bcbe543aa4c8230aeb5b4e071d" gracePeriod=30 Jul 09 09:45:48.499138 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:48.499074 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7c964dbd44-rn899" podUID="bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc" containerName="kube-rbac-proxy" containerID="cri-o://30c4a9b7c5593cdc5225e3df859b15a2e55c66160864e5f573387063efa1862b" gracePeriod=30 Jul 09 09:45:49.488306 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:49.488271 2578 generic.go:356] "Generic (PLEG): container finished" podID="953cb958-4294-4c91-912a-3d4ac0bf4c67" containerID="d3039f4e9adabdc9905284da36df318dd2bdfad8d441f28c9b0981c252093442" exitCode=2 Jul 09 09:45:49.488733 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:49.488357 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-4cp66" event={"ID":"953cb958-4294-4c91-912a-3d4ac0bf4c67","Type":"ContainerDied","Data":"d3039f4e9adabdc9905284da36df318dd2bdfad8d441f28c9b0981c252093442"} Jul 09 09:45:49.490725 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:49.490698 2578 generic.go:356] "Generic (PLEG): container finished" podID="bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc" containerID="30c4a9b7c5593cdc5225e3df859b15a2e55c66160864e5f573387063efa1862b" exitCode=2 Jul 09 09:45:49.490880 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:49.490734 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7c964dbd44-rn899" event={"ID":"bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc","Type":"ContainerDied","Data":"30c4a9b7c5593cdc5225e3df859b15a2e55c66160864e5f573387063efa1862b"} Jul 09 09:45:49.492646 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:49.492626 2578 generic.go:356] "Generic (PLEG): container finished" podID="c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca" containerID="ccbe677866490ccb27824b096764b8585344631c045eb531c3952b03ad27a7bb" exitCode=2 Jul 09 09:45:49.492749 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:49.492691 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-84b568d556-46fps" event={"ID":"c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca","Type":"ContainerDied","Data":"ccbe677866490ccb27824b096764b8585344631c045eb531c3952b03ad27a7bb"} Jul 09 09:45:51.025256 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:51.025219 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-4cp66" podUID="953cb958-4294-4c91-912a-3d4ac0bf4c67" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.24:8643/healthz\": dial tcp 10.133.0.24:8643: connect: connection refused" Jul 09 09:45:51.025627 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:51.025219 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7c964dbd44-rn899" podUID="bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.23:8643/healthz\": dial tcp 10.133.0.23:8643: connect: connection refused" Jul 09 09:45:51.029631 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:51.029602 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-4cp66" podUID="953cb958-4294-4c91-912a-3d4ac0bf4c67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused" Jul 09 09:45:51.029743 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:51.029606 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7c964dbd44-rn899" podUID="bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused" Jul 09 09:45:51.853274 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:51.853246 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-4cp66" Jul 09 09:45:51.919789 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:51.919727 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/953cb958-4294-4c91-912a-3d4ac0bf4c67-kube-rbac-proxy-sar-config\") pod \"953cb958-4294-4c91-912a-3d4ac0bf4c67\" (UID: \"953cb958-4294-4c91-912a-3d4ac0bf4c67\") " Jul 09 09:45:51.919937 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:51.919787 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/953cb958-4294-4c91-912a-3d4ac0bf4c67-cabundle-cert\") pod \"953cb958-4294-4c91-912a-3d4ac0bf4c67\" (UID: \"953cb958-4294-4c91-912a-3d4ac0bf4c67\") " Jul 09 09:45:51.919937 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:51.919874 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sq8gx\" (UniqueName: \"kubernetes.io/projected/953cb958-4294-4c91-912a-3d4ac0bf4c67-kube-api-access-sq8gx\") pod \"953cb958-4294-4c91-912a-3d4ac0bf4c67\" (UID: \"953cb958-4294-4c91-912a-3d4ac0bf4c67\") " Jul 09 09:45:51.919937 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:51.919926 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/953cb958-4294-4c91-912a-3d4ac0bf4c67-proxy-tls\") pod \"953cb958-4294-4c91-912a-3d4ac0bf4c67\" (UID: \"953cb958-4294-4c91-912a-3d4ac0bf4c67\") " Jul 09 09:45:51.920097 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:51.919995 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/953cb958-4294-4c91-912a-3d4ac0bf4c67-kserve-provision-location\") pod \"953cb958-4294-4c91-912a-3d4ac0bf4c67\" (UID: \"953cb958-4294-4c91-912a-3d4ac0bf4c67\") " Jul 09 09:45:51.920145 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:51.920098 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/953cb958-4294-4c91-912a-3d4ac0bf4c67-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "953cb958-4294-4c91-912a-3d4ac0bf4c67" (UID: "953cb958-4294-4c91-912a-3d4ac0bf4c67"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 09:45:51.920145 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:51.920129 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/953cb958-4294-4c91-912a-3d4ac0bf4c67-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "953cb958-4294-4c91-912a-3d4ac0bf4c67" (UID: "953cb958-4294-4c91-912a-3d4ac0bf4c67"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 09:45:51.920300 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:51.920278 2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/953cb958-4294-4c91-912a-3d4ac0bf4c67-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-204.ec2.internal\" DevicePath \"\"" Jul 09 09:45:51.920363 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:51.920306 2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/953cb958-4294-4c91-912a-3d4ac0bf4c67-cabundle-cert\") on node \"ip-10-0-135-204.ec2.internal\" DevicePath \"\"" Jul 09 09:45:51.920363 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:51.920331 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/953cb958-4294-4c91-912a-3d4ac0bf4c67-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "953cb958-4294-4c91-912a-3d4ac0bf4c67" (UID: "953cb958-4294-4c91-912a-3d4ac0bf4c67"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 09:45:51.922322 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:51.922292 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/953cb958-4294-4c91-912a-3d4ac0bf4c67-kube-api-access-sq8gx" (OuterVolumeSpecName: "kube-api-access-sq8gx") pod "953cb958-4294-4c91-912a-3d4ac0bf4c67" (UID: "953cb958-4294-4c91-912a-3d4ac0bf4c67"). InnerVolumeSpecName "kube-api-access-sq8gx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 09:45:51.922322 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:51.922308 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/953cb958-4294-4c91-912a-3d4ac0bf4c67-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "953cb958-4294-4c91-912a-3d4ac0bf4c67" (UID: "953cb958-4294-4c91-912a-3d4ac0bf4c67"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 09:45:52.021194 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:52.021169 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sq8gx\" (UniqueName: \"kubernetes.io/projected/953cb958-4294-4c91-912a-3d4ac0bf4c67-kube-api-access-sq8gx\") on node \"ip-10-0-135-204.ec2.internal\" DevicePath \"\"" Jul 09 09:45:52.021194 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:52.021191 2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/953cb958-4294-4c91-912a-3d4ac0bf4c67-proxy-tls\") on node \"ip-10-0-135-204.ec2.internal\" DevicePath \"\"" Jul 09 09:45:52.021194 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:52.021201 2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/953cb958-4294-4c91-912a-3d4ac0bf4c67-kserve-provision-location\") on node \"ip-10-0-135-204.ec2.internal\" DevicePath \"\"" Jul 09 09:45:52.028738 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:52.028714 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-84b568d556-46fps" podUID="c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused" Jul 09 09:45:52.033127 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:52.033107 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-84b568d556-46fps" podUID="c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused" Jul 09 09:45:52.507341 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:52.507307 2578 generic.go:356] "Generic (PLEG): container finished" podID="953cb958-4294-4c91-912a-3d4ac0bf4c67" containerID="699676e75c2a0c2b92b1d769beb5633e5ba576fb802cd252f8db661feee3ee7b" exitCode=0 Jul 09 09:45:52.507506 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:52.507375 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-4cp66" Jul 09 09:45:52.507506 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:52.507389 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-4cp66" event={"ID":"953cb958-4294-4c91-912a-3d4ac0bf4c67","Type":"ContainerDied","Data":"699676e75c2a0c2b92b1d769beb5633e5ba576fb802cd252f8db661feee3ee7b"} Jul 09 09:45:52.507506 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:52.507448 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-4cp66" event={"ID":"953cb958-4294-4c91-912a-3d4ac0bf4c67","Type":"ContainerDied","Data":"1c2a6934f750f923987608a79ec39f8471827ccc3e1b1e4d881c6e41c13277ad"} Jul 09 09:45:52.507506 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:52.507474 2578 scope.go:117] "RemoveContainer" containerID="d3039f4e9adabdc9905284da36df318dd2bdfad8d441f28c9b0981c252093442" Jul 09 09:45:52.515971 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:52.515953 2578 scope.go:117] "RemoveContainer" containerID="699676e75c2a0c2b92b1d769beb5633e5ba576fb802cd252f8db661feee3ee7b" Jul 09 09:45:52.523290 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:52.523272 2578 scope.go:117] "RemoveContainer" containerID="01132d0ee444236160f5e6d4681915ada5c540f9f7887bb31e3dfda4515de0a1" Jul 09 09:45:52.531165 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:52.531147 2578 scope.go:117] "RemoveContainer" containerID="d3039f4e9adabdc9905284da36df318dd2bdfad8d441f28c9b0981c252093442" Jul 09 09:45:52.531436 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:45:52.531418 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3039f4e9adabdc9905284da36df318dd2bdfad8d441f28c9b0981c252093442\": container with ID starting with d3039f4e9adabdc9905284da36df318dd2bdfad8d441f28c9b0981c252093442 not found: ID does not exist" containerID="d3039f4e9adabdc9905284da36df318dd2bdfad8d441f28c9b0981c252093442" Jul 09 09:45:52.531511 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:52.531447 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3039f4e9adabdc9905284da36df318dd2bdfad8d441f28c9b0981c252093442"} err="failed to get container status \"d3039f4e9adabdc9905284da36df318dd2bdfad8d441f28c9b0981c252093442\": rpc error: code = NotFound desc = could not find container \"d3039f4e9adabdc9905284da36df318dd2bdfad8d441f28c9b0981c252093442\": container with ID starting with d3039f4e9adabdc9905284da36df318dd2bdfad8d441f28c9b0981c252093442 not found: ID does not exist" Jul 09 09:45:52.531511 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:52.531470 2578 scope.go:117] "RemoveContainer" containerID="699676e75c2a0c2b92b1d769beb5633e5ba576fb802cd252f8db661feee3ee7b" Jul 09 09:45:52.531763 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:45:52.531746 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"699676e75c2a0c2b92b1d769beb5633e5ba576fb802cd252f8db661feee3ee7b\": container with ID starting with 699676e75c2a0c2b92b1d769beb5633e5ba576fb802cd252f8db661feee3ee7b not found: ID does not exist" containerID="699676e75c2a0c2b92b1d769beb5633e5ba576fb802cd252f8db661feee3ee7b" Jul 09 09:45:52.531881 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:52.531770 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"699676e75c2a0c2b92b1d769beb5633e5ba576fb802cd252f8db661feee3ee7b"} err="failed to get container status \"699676e75c2a0c2b92b1d769beb5633e5ba576fb802cd252f8db661feee3ee7b\": rpc error: code = NotFound desc = could not find container \"699676e75c2a0c2b92b1d769beb5633e5ba576fb802cd252f8db661feee3ee7b\": container with ID starting with 699676e75c2a0c2b92b1d769beb5633e5ba576fb802cd252f8db661feee3ee7b not found: ID does not exist" Jul 09 09:45:52.531881 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:52.531790 2578 scope.go:117] "RemoveContainer" containerID="01132d0ee444236160f5e6d4681915ada5c540f9f7887bb31e3dfda4515de0a1" Jul 09 09:45:52.532140 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:45:52.532113 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01132d0ee444236160f5e6d4681915ada5c540f9f7887bb31e3dfda4515de0a1\": container with ID starting with 01132d0ee444236160f5e6d4681915ada5c540f9f7887bb31e3dfda4515de0a1 not found: ID does not exist" containerID="01132d0ee444236160f5e6d4681915ada5c540f9f7887bb31e3dfda4515de0a1" Jul 09 09:45:52.532196 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:52.532137 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01132d0ee444236160f5e6d4681915ada5c540f9f7887bb31e3dfda4515de0a1"} err="failed to get container status \"01132d0ee444236160f5e6d4681915ada5c540f9f7887bb31e3dfda4515de0a1\": rpc error: code = NotFound desc = could not find container \"01132d0ee444236160f5e6d4681915ada5c540f9f7887bb31e3dfda4515de0a1\": container with ID starting with 01132d0ee444236160f5e6d4681915ada5c540f9f7887bb31e3dfda4515de0a1 not found: ID does not exist" Jul 09 09:45:52.532817 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:52.532799 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-4cp66"] Jul 09 09:45:52.536396 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:52.536373 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-4cp66"] Jul 09 09:45:53.195390 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:53.195372 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7c964dbd44-rn899" Jul 09 09:45:53.205954 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:53.205934 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-84b568d556-46fps" Jul 09 09:45:53.329620 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:53.329588 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-69g6k\" (UniqueName: \"kubernetes.io/projected/bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc-kube-api-access-69g6k\") pod \"bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc\" (UID: \"bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc\") " Jul 09 09:45:53.329773 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:53.329630 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc-kserve-provision-location\") pod \"bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc\" (UID: \"bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc\") " Jul 09 09:45:53.329773 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:53.329652 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca-cabundle-cert\") pod \"c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca\" (UID: \"c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca\") " Jul 09 09:45:53.329773 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:53.329684 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc-proxy-tls\") pod \"bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc\" (UID: \"bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc\") " Jul 09 09:45:53.329773 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:53.329701 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc-cabundle-cert\") pod \"bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc\" (UID: \"bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc\") " Jul 09 09:45:53.329773 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:53.329742 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc-kube-rbac-proxy-sar-config\") pod \"bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc\" (UID: \"bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc\") " Jul 09 09:45:53.330042 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:53.329957 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc" (UID: "bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 09:45:53.330094 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:53.330051 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca-kserve-provision-location\") pod \"c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca\" (UID: \"c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca\") " Jul 09 09:45:53.330151 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:53.330087 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc" (UID: "bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 09:45:53.330151 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:53.330097 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca" (UID: "c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 09:45:53.330151 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:53.330124 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j6scc\" (UniqueName: \"kubernetes.io/projected/c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca-kube-api-access-j6scc\") pod \"c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca\" (UID: \"c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca\") " Jul 09 09:45:53.330151 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:53.330137 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc" (UID: "bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 09:45:53.330332 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:53.330152 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca-proxy-tls\") pod \"c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca\" (UID: \"c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca\") " Jul 09 09:45:53.330332 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:53.330179 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca-kube-rbac-proxy-sar-config\") pod \"c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca\" (UID: \"c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca\") " Jul 09 09:45:53.330421 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:53.330363 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca" (UID: "c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 09:45:53.330638 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:53.330611 2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-204.ec2.internal\" DevicePath \"\"" Jul 09 09:45:53.330638 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:53.330635 2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca-kserve-provision-location\") on node \"ip-10-0-135-204.ec2.internal\" DevicePath \"\"" Jul 09 09:45:53.330758 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:53.330650 2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc-kserve-provision-location\") on node \"ip-10-0-135-204.ec2.internal\" DevicePath \"\"" Jul 09 09:45:53.330758 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:53.330664 2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca-cabundle-cert\") on node \"ip-10-0-135-204.ec2.internal\" DevicePath \"\"" Jul 09 09:45:53.330758 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:53.330668 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca" (UID: "c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 09:45:53.330758 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:53.330679 2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc-cabundle-cert\") on node \"ip-10-0-135-204.ec2.internal\" DevicePath \"\"" Jul 09 09:45:53.332046 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:53.332018 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc" (UID: "bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 09:45:53.332224 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:53.332203 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca" (UID: "c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 09:45:53.332437 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:53.332418 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc-kube-api-access-69g6k" (OuterVolumeSpecName: "kube-api-access-69g6k") pod "bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc" (UID: "bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc"). InnerVolumeSpecName "kube-api-access-69g6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 09:45:53.332608 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:53.332587 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca-kube-api-access-j6scc" (OuterVolumeSpecName: "kube-api-access-j6scc") pod "c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca" (UID: "c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca"). InnerVolumeSpecName "kube-api-access-j6scc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 09:45:53.431808 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:53.431770 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69g6k\" (UniqueName: \"kubernetes.io/projected/bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc-kube-api-access-69g6k\") on node \"ip-10-0-135-204.ec2.internal\" DevicePath \"\"" Jul 09 09:45:53.431808 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:53.431804 2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc-proxy-tls\") on node \"ip-10-0-135-204.ec2.internal\" DevicePath \"\"" Jul 09 09:45:53.431808 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:53.431815 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6scc\" (UniqueName: \"kubernetes.io/projected/c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca-kube-api-access-j6scc\") on node \"ip-10-0-135-204.ec2.internal\" DevicePath \"\"" Jul 09 09:45:53.432016 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:53.431823 2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca-proxy-tls\") on node \"ip-10-0-135-204.ec2.internal\" DevicePath \"\"" Jul 09 09:45:53.432016 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:53.431831 2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-204.ec2.internal\" DevicePath \"\"" Jul 09 09:45:53.442677 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:53.442647 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="953cb958-4294-4c91-912a-3d4ac0bf4c67" path="/var/lib/kubelet/pods/953cb958-4294-4c91-912a-3d4ac0bf4c67/volumes" Jul 09 09:45:53.514139 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:53.514115 2578 generic.go:356] "Generic (PLEG): container finished" podID="bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc" containerID="1740f57cbe1cb96fba0c509be76bc8b8bbe754bcbe543aa4c8230aeb5b4e071d" exitCode=0 Jul 09 09:45:53.514236 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:53.514180 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7c964dbd44-rn899" Jul 09 09:45:53.514236 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:53.514196 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7c964dbd44-rn899" event={"ID":"bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc","Type":"ContainerDied","Data":"1740f57cbe1cb96fba0c509be76bc8b8bbe754bcbe543aa4c8230aeb5b4e071d"} Jul 09 09:45:53.514364 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:53.514276 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7c964dbd44-rn899" event={"ID":"bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc","Type":"ContainerDied","Data":"b4adb1525adce6934a5a3251031e0b99043cd9173b92a8b4e8f8675b2eca4561"} Jul 09 09:45:53.514364 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:53.514317 2578 scope.go:117] "RemoveContainer" containerID="30c4a9b7c5593cdc5225e3df859b15a2e55c66160864e5f573387063efa1862b" Jul 09 09:45:53.516386 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:53.516361 2578 generic.go:356] "Generic (PLEG): container finished" podID="c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca" containerID="0e1b7c13bb308a3c00c2e2dcf157ed9f9f82ca8a33d3a33945fef17058d1d3ec" exitCode=0 Jul 09 09:45:53.516507 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:53.516416 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-84b568d556-46fps" event={"ID":"c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca","Type":"ContainerDied","Data":"0e1b7c13bb308a3c00c2e2dcf157ed9f9f82ca8a33d3a33945fef17058d1d3ec"} Jul 09 09:45:53.516560 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:53.516422 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-84b568d556-46fps" Jul 09 09:45:53.516643 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:53.516510 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-84b568d556-46fps" event={"ID":"c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca","Type":"ContainerDied","Data":"f9a2ed1c63b420afb434b6025b4e6d199e86bc2b2354ac74d01736a1d3cdbbaf"} Jul 09 09:45:53.522688 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:53.522673 2578 scope.go:117] "RemoveContainer" containerID="1740f57cbe1cb96fba0c509be76bc8b8bbe754bcbe543aa4c8230aeb5b4e071d" Jul 09 09:45:53.529753 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:53.529722 2578 scope.go:117] "RemoveContainer" containerID="10383cd652304340bdf01c8a577118c47eb953458ee645770927592f7fe10630" Jul 09 09:45:53.534795 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:53.534775 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7c964dbd44-rn899"] Jul 09 09:45:53.536938 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:53.536926 2578 scope.go:117] "RemoveContainer" containerID="30c4a9b7c5593cdc5225e3df859b15a2e55c66160864e5f573387063efa1862b" Jul 09 09:45:53.537166 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:45:53.537148 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30c4a9b7c5593cdc5225e3df859b15a2e55c66160864e5f573387063efa1862b\": container with ID starting with 30c4a9b7c5593cdc5225e3df859b15a2e55c66160864e5f573387063efa1862b not found: ID does not exist" containerID="30c4a9b7c5593cdc5225e3df859b15a2e55c66160864e5f573387063efa1862b" Jul 09 09:45:53.537215 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:53.537174 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30c4a9b7c5593cdc5225e3df859b15a2e55c66160864e5f573387063efa1862b"} err="failed to get container status \"30c4a9b7c5593cdc5225e3df859b15a2e55c66160864e5f573387063efa1862b\": rpc error: code = NotFound desc = could not find container \"30c4a9b7c5593cdc5225e3df859b15a2e55c66160864e5f573387063efa1862b\": container with ID starting with 30c4a9b7c5593cdc5225e3df859b15a2e55c66160864e5f573387063efa1862b not found: ID does not exist" Jul 09 09:45:53.537215 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:53.537190 2578 scope.go:117] "RemoveContainer" containerID="1740f57cbe1cb96fba0c509be76bc8b8bbe754bcbe543aa4c8230aeb5b4e071d" Jul 09 09:45:53.537404 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:45:53.537385 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1740f57cbe1cb96fba0c509be76bc8b8bbe754bcbe543aa4c8230aeb5b4e071d\": container with ID starting with 1740f57cbe1cb96fba0c509be76bc8b8bbe754bcbe543aa4c8230aeb5b4e071d not found: ID does not exist" containerID="1740f57cbe1cb96fba0c509be76bc8b8bbe754bcbe543aa4c8230aeb5b4e071d" Jul 09 09:45:53.537463 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:53.537415 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1740f57cbe1cb96fba0c509be76bc8b8bbe754bcbe543aa4c8230aeb5b4e071d"} err="failed to get container status \"1740f57cbe1cb96fba0c509be76bc8b8bbe754bcbe543aa4c8230aeb5b4e071d\": rpc error: code = NotFound desc = could not find container \"1740f57cbe1cb96fba0c509be76bc8b8bbe754bcbe543aa4c8230aeb5b4e071d\": container with ID starting with 1740f57cbe1cb96fba0c509be76bc8b8bbe754bcbe543aa4c8230aeb5b4e071d not found: ID does not exist" Jul 09 09:45:53.537463 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:53.537439 2578 scope.go:117] "RemoveContainer" containerID="10383cd652304340bdf01c8a577118c47eb953458ee645770927592f7fe10630" Jul 09 09:45:53.537655 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:45:53.537640 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10383cd652304340bdf01c8a577118c47eb953458ee645770927592f7fe10630\": container with ID starting with 10383cd652304340bdf01c8a577118c47eb953458ee645770927592f7fe10630 not found: ID does not exist" containerID="10383cd652304340bdf01c8a577118c47eb953458ee645770927592f7fe10630" Jul 09 09:45:53.537713 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:53.537662 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10383cd652304340bdf01c8a577118c47eb953458ee645770927592f7fe10630"} err="failed to get container status \"10383cd652304340bdf01c8a577118c47eb953458ee645770927592f7fe10630\": rpc error: code = NotFound desc = could not find container \"10383cd652304340bdf01c8a577118c47eb953458ee645770927592f7fe10630\": container with ID starting with 10383cd652304340bdf01c8a577118c47eb953458ee645770927592f7fe10630 not found: ID does not exist" Jul 09 09:45:53.537713 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:53.537680 2578 scope.go:117] "RemoveContainer" containerID="ccbe677866490ccb27824b096764b8585344631c045eb531c3952b03ad27a7bb" Jul 09 09:45:53.540587 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:53.540569 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7c964dbd44-rn899"] Jul 09 09:45:53.545186 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:53.545172 2578 scope.go:117] "RemoveContainer" containerID="0e1b7c13bb308a3c00c2e2dcf157ed9f9f82ca8a33d3a33945fef17058d1d3ec" Jul 09 09:45:53.551388 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:53.551367 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-84b568d556-46fps"] Jul 09 09:45:53.552721 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:53.552699 2578 scope.go:117] "RemoveContainer" containerID="84f18c9e86e6ac3fe35d868f21cc9844d744b12150620f1f0184f8c43a1fa1a5" Jul 09 09:45:53.554985 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:53.554967 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-84b568d556-46fps"] Jul 09 09:45:53.559954 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:53.559938 2578 scope.go:117] "RemoveContainer" containerID="ccbe677866490ccb27824b096764b8585344631c045eb531c3952b03ad27a7bb" Jul 09 09:45:53.560201 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:45:53.560185 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccbe677866490ccb27824b096764b8585344631c045eb531c3952b03ad27a7bb\": container with ID starting with ccbe677866490ccb27824b096764b8585344631c045eb531c3952b03ad27a7bb not found: ID does not exist" containerID="ccbe677866490ccb27824b096764b8585344631c045eb531c3952b03ad27a7bb" Jul 09 09:45:53.560248 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:53.560206 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccbe677866490ccb27824b096764b8585344631c045eb531c3952b03ad27a7bb"} err="failed to get container status \"ccbe677866490ccb27824b096764b8585344631c045eb531c3952b03ad27a7bb\": rpc error: code = NotFound desc = could not find container \"ccbe677866490ccb27824b096764b8585344631c045eb531c3952b03ad27a7bb\": container with ID starting with ccbe677866490ccb27824b096764b8585344631c045eb531c3952b03ad27a7bb not found: ID does not exist" Jul 09 09:45:53.560248 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:53.560221 2578 scope.go:117] "RemoveContainer" containerID="0e1b7c13bb308a3c00c2e2dcf157ed9f9f82ca8a33d3a33945fef17058d1d3ec" Jul 09 09:45:53.560422 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:45:53.560406 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e1b7c13bb308a3c00c2e2dcf157ed9f9f82ca8a33d3a33945fef17058d1d3ec\": container with ID starting with 0e1b7c13bb308a3c00c2e2dcf157ed9f9f82ca8a33d3a33945fef17058d1d3ec not found: ID does not exist" containerID="0e1b7c13bb308a3c00c2e2dcf157ed9f9f82ca8a33d3a33945fef17058d1d3ec" Jul 09 09:45:53.560457 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:53.560428 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e1b7c13bb308a3c00c2e2dcf157ed9f9f82ca8a33d3a33945fef17058d1d3ec"} err="failed to get container status \"0e1b7c13bb308a3c00c2e2dcf157ed9f9f82ca8a33d3a33945fef17058d1d3ec\": rpc error: code = NotFound desc = could not find container \"0e1b7c13bb308a3c00c2e2dcf157ed9f9f82ca8a33d3a33945fef17058d1d3ec\": container with ID starting with 0e1b7c13bb308a3c00c2e2dcf157ed9f9f82ca8a33d3a33945fef17058d1d3ec not found: ID does not exist" Jul 09 09:45:53.560457 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:53.560444 2578 scope.go:117] "RemoveContainer" containerID="84f18c9e86e6ac3fe35d868f21cc9844d744b12150620f1f0184f8c43a1fa1a5" Jul 09 09:45:53.560680 ip-10-0-135-204 kubenswrapper[2578]: E0709 09:45:53.560665 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84f18c9e86e6ac3fe35d868f21cc9844d744b12150620f1f0184f8c43a1fa1a5\": container with ID starting with 84f18c9e86e6ac3fe35d868f21cc9844d744b12150620f1f0184f8c43a1fa1a5 not found: ID does not exist" containerID="84f18c9e86e6ac3fe35d868f21cc9844d744b12150620f1f0184f8c43a1fa1a5" Jul 09 09:45:53.560733 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:53.560682 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84f18c9e86e6ac3fe35d868f21cc9844d744b12150620f1f0184f8c43a1fa1a5"} err="failed to get container status \"84f18c9e86e6ac3fe35d868f21cc9844d744b12150620f1f0184f8c43a1fa1a5\": rpc error: code = NotFound desc = could not find container \"84f18c9e86e6ac3fe35d868f21cc9844d744b12150620f1f0184f8c43a1fa1a5\": container with ID starting with 84f18c9e86e6ac3fe35d868f21cc9844d744b12150620f1f0184f8c43a1fa1a5 not found: ID does not exist" Jul 09 09:45:55.442280 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:55.442249 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc" path="/var/lib/kubelet/pods/bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc/volumes" Jul 09 09:45:55.442714 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:45:55.442701 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca" path="/var/lib/kubelet/pods/c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca/volumes" Jul 09 09:48:51.712456 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:48:51.712427 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xf4sw_1677420d-7376-45ad-ac67-20128aec5756/ovn-acl-logging/0.log" Jul 09 09:48:51.714622 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:48:51.714598 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xf4sw_1677420d-7376-45ad-ac67-20128aec5756/ovn-acl-logging/0.log" Jul 09 09:53:51.735792 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:53:51.735767 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xf4sw_1677420d-7376-45ad-ac67-20128aec5756/ovn-acl-logging/0.log" Jul 09 09:53:51.738145 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:53:51.738128 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xf4sw_1677420d-7376-45ad-ac67-20128aec5756/ovn-acl-logging/0.log" Jul 09 09:58:51.758606 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:58:51.758501 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xf4sw_1677420d-7376-45ad-ac67-20128aec5756/ovn-acl-logging/0.log" Jul 09 09:58:51.762240 ip-10-0-135-204 kubenswrapper[2578]: I0709 09:58:51.761951 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xf4sw_1677420d-7376-45ad-ac67-20128aec5756/ovn-acl-logging/0.log" Jul 09 10:03:51.786630 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:03:51.786524 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xf4sw_1677420d-7376-45ad-ac67-20128aec5756/ovn-acl-logging/0.log" Jul 09 10:03:51.790146 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:03:51.790126 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xf4sw_1677420d-7376-45ad-ac67-20128aec5756/ovn-acl-logging/0.log" Jul 09 10:08:51.809350 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:08:51.809243 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xf4sw_1677420d-7376-45ad-ac67-20128aec5756/ovn-acl-logging/0.log" Jul 09 10:08:51.813129 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:08:51.813114 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xf4sw_1677420d-7376-45ad-ac67-20128aec5756/ovn-acl-logging/0.log" Jul 09 10:13:51.840088 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:13:51.839986 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xf4sw_1677420d-7376-45ad-ac67-20128aec5756/ovn-acl-logging/0.log" Jul 09 10:13:51.843405 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:13:51.843054 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xf4sw_1677420d-7376-45ad-ac67-20128aec5756/ovn-acl-logging/0.log" Jul 09 10:18:51.862901 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:18:51.862772 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xf4sw_1677420d-7376-45ad-ac67-20128aec5756/ovn-acl-logging/0.log" Jul 09 10:18:51.866997 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:18:51.866980 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xf4sw_1677420d-7376-45ad-ac67-20128aec5756/ovn-acl-logging/0.log" Jul 09 10:22:48.873510 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:22:48.873477 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9qx4b/must-gather-v4shm"] Jul 09 10:22:48.873987 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:22:48.873777 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="953cb958-4294-4c91-912a-3d4ac0bf4c67" containerName="kserve-container" Jul 09 10:22:48.873987 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:22:48.873787 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="953cb958-4294-4c91-912a-3d4ac0bf4c67" containerName="kserve-container" Jul 09 10:22:48.873987 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:22:48.873798 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca" containerName="storage-initializer" Jul 09 10:22:48.873987 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:22:48.873803 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca" containerName="storage-initializer" Jul 09 10:22:48.873987 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:22:48.873810 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca" containerName="kserve-container" Jul 09 10:22:48.873987 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:22:48.873815 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca" containerName="kserve-container" Jul 09 10:22:48.873987 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:22:48.873822 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="953cb958-4294-4c91-912a-3d4ac0bf4c67" containerName="storage-initializer" Jul 09 10:22:48.873987 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:22:48.873827 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="953cb958-4294-4c91-912a-3d4ac0bf4c67" containerName="storage-initializer" Jul 09 10:22:48.873987 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:22:48.873835 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="953cb958-4294-4c91-912a-3d4ac0bf4c67" containerName="kube-rbac-proxy" Jul 09 10:22:48.873987 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:22:48.873856 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="953cb958-4294-4c91-912a-3d4ac0bf4c67" containerName="kube-rbac-proxy" Jul 09 10:22:48.873987 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:22:48.873874 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc" containerName="storage-initializer" Jul 09 10:22:48.873987 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:22:48.873883 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc" containerName="storage-initializer" Jul 09 10:22:48.873987 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:22:48.873893 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc" containerName="kserve-container" Jul 09 10:22:48.873987 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:22:48.873901 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc" containerName="kserve-container" Jul 09 10:22:48.873987 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:22:48.873909 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc" containerName="kube-rbac-proxy" Jul 09 10:22:48.873987 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:22:48.873914 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc" containerName="kube-rbac-proxy" Jul 09 10:22:48.873987 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:22:48.873920 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca" containerName="kube-rbac-proxy" Jul 09 10:22:48.873987 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:22:48.873925 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca" containerName="kube-rbac-proxy" Jul 09 10:22:48.873987 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:22:48.873973 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="953cb958-4294-4c91-912a-3d4ac0bf4c67" containerName="kube-rbac-proxy" Jul 09 10:22:48.873987 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:22:48.873980 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca" containerName="kserve-container" Jul 09 10:22:48.873987 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:22:48.873986 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="953cb958-4294-4c91-912a-3d4ac0bf4c67" containerName="kserve-container" Jul 09 10:22:48.873987 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:22:48.873994 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8d2c03e-cf8b-4bcc-b354-11e93e4b3bca" containerName="kube-rbac-proxy" Jul 09 10:22:48.873987 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:22:48.873999 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc" containerName="kserve-container" Jul 09 10:22:48.874670 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:22:48.874004 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd442d1d-ea1c-47cd-9b75-7f6eb7f61cbc" containerName="kube-rbac-proxy" Jul 09 10:22:48.877022 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:22:48.877006 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9qx4b/must-gather-v4shm" Jul 09 10:22:48.880752 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:22:48.880734 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9qx4b\"/\"default-dockercfg-t42pg\"" Jul 09 10:22:48.881465 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:22:48.881446 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9qx4b\"/\"openshift-service-ca.crt\"" Jul 09 10:22:48.881565 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:22:48.881474 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9qx4b\"/\"kube-root-ca.crt\"" Jul 09 10:22:48.889887 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:22:48.889865 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9qx4b/must-gather-v4shm"] Jul 09 10:22:48.944089 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:22:48.944061 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/aefde989-67b4-44c1-816d-fa8ea22048bc-must-gather-output\") pod \"must-gather-v4shm\" (UID: \"aefde989-67b4-44c1-816d-fa8ea22048bc\") " pod="openshift-must-gather-9qx4b/must-gather-v4shm" Jul 09 10:22:48.944180 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:22:48.944102 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnzms\" (UniqueName: \"kubernetes.io/projected/aefde989-67b4-44c1-816d-fa8ea22048bc-kube-api-access-gnzms\") pod \"must-gather-v4shm\" (UID: \"aefde989-67b4-44c1-816d-fa8ea22048bc\") " pod="openshift-must-gather-9qx4b/must-gather-v4shm" Jul 09 10:22:49.044589 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:22:49.044565 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/aefde989-67b4-44c1-816d-fa8ea22048bc-must-gather-output\") pod \"must-gather-v4shm\" (UID: \"aefde989-67b4-44c1-816d-fa8ea22048bc\") " pod="openshift-must-gather-9qx4b/must-gather-v4shm" Jul 09 10:22:49.044706 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:22:49.044614 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnzms\" (UniqueName: \"kubernetes.io/projected/aefde989-67b4-44c1-816d-fa8ea22048bc-kube-api-access-gnzms\") pod \"must-gather-v4shm\" (UID: \"aefde989-67b4-44c1-816d-fa8ea22048bc\") " pod="openshift-must-gather-9qx4b/must-gather-v4shm" Jul 09 10:22:49.044962 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:22:49.044937 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/aefde989-67b4-44c1-816d-fa8ea22048bc-must-gather-output\") pod \"must-gather-v4shm\" (UID: \"aefde989-67b4-44c1-816d-fa8ea22048bc\") " pod="openshift-must-gather-9qx4b/must-gather-v4shm" Jul 09 10:22:49.052832 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:22:49.052805 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnzms\" (UniqueName: \"kubernetes.io/projected/aefde989-67b4-44c1-816d-fa8ea22048bc-kube-api-access-gnzms\") pod \"must-gather-v4shm\" (UID: \"aefde989-67b4-44c1-816d-fa8ea22048bc\") " pod="openshift-must-gather-9qx4b/must-gather-v4shm" Jul 09 10:22:49.198939 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:22:49.198912 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9qx4b/must-gather-v4shm" Jul 09 10:22:49.325644 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:22:49.325610 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9qx4b/must-gather-v4shm"] Jul 09 10:22:49.328515 ip-10-0-135-204 kubenswrapper[2578]: W0709 10:22:49.328488 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaefde989_67b4_44c1_816d_fa8ea22048bc.slice/crio-0b2cda45f457c9da470140877309c30cb794292907162c47ae7484f5a1c7ddff WatchSource:0}: Error finding container 0b2cda45f457c9da470140877309c30cb794292907162c47ae7484f5a1c7ddff: Status 404 returned error can't find the container with id 0b2cda45f457c9da470140877309c30cb794292907162c47ae7484f5a1c7ddff Jul 09 10:22:49.330198 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:22:49.330180 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 10:22:49.762777 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:22:49.762745 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9qx4b/must-gather-v4shm" event={"ID":"aefde989-67b4-44c1-816d-fa8ea22048bc","Type":"ContainerStarted","Data":"0b2cda45f457c9da470140877309c30cb794292907162c47ae7484f5a1c7ddff"} Jul 09 10:22:55.794448 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:22:55.794411 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9qx4b/must-gather-v4shm" event={"ID":"aefde989-67b4-44c1-816d-fa8ea22048bc","Type":"ContainerStarted","Data":"e6217852884a804af15234f023f97a77c7bc14a77731e829d9a527778c1bf349"} Jul 09 10:22:55.794862 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:22:55.794460 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9qx4b/must-gather-v4shm" event={"ID":"aefde989-67b4-44c1-816d-fa8ea22048bc","Type":"ContainerStarted","Data":"ab751f674480eb5fe1f84850d61fb0a3f59ea413d1648594ab21235a384e5e00"} Jul 09 10:22:55.814409 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:22:55.814348 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9qx4b/must-gather-v4shm" podStartSLOduration=1.690696168 podStartE2EDuration="7.814327811s" totalImagesPullingTime="6.123631643s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:22:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:22:55.81228958 +0000 UTC m=+2960.904043502" watchObservedRunningTime="2026-07-09 10:22:55.814327811 +0000 UTC m=+2960.906081732" Jul 09 10:23:19.897709 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:19.897678 2578 generic.go:356] "Generic (PLEG): container finished" podID="aefde989-67b4-44c1-816d-fa8ea22048bc" containerID="ab751f674480eb5fe1f84850d61fb0a3f59ea413d1648594ab21235a384e5e00" exitCode=0 Jul 09 10:23:19.898117 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:19.897750 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9qx4b/must-gather-v4shm" event={"ID":"aefde989-67b4-44c1-816d-fa8ea22048bc","Type":"ContainerDied","Data":"ab751f674480eb5fe1f84850d61fb0a3f59ea413d1648594ab21235a384e5e00"} Jul 09 10:23:19.898117 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:19.898039 2578 scope.go:117] "RemoveContainer" containerID="ab751f674480eb5fe1f84850d61fb0a3f59ea413d1648594ab21235a384e5e00" Jul 09 10:23:19.931459 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:19.931433 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-9qx4b_must-gather-v4shm_aefde989-67b4-44c1-816d-fa8ea22048bc/gather/0.log" Jul 09 10:23:24.460222 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:24.460192 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-sw9zq_a0c24dc3-5e21-4537-966b-0c5f9e46f8d9/global-pull-secret-syncer/0.log" Jul 09 10:23:24.560210 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:24.560185 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-gbmbr_dc248a4e-4aa6-4ea4-81b7-439e669b56c1/konnectivity-agent/0.log" Jul 09 10:23:24.621439 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:24.621414 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-204.ec2.internal_4386692e678e561186646b99422b104a/haproxy/0.log" Jul 09 10:23:25.359047 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:25.359017 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-9qx4b/must-gather-v4shm"] Jul 09 10:23:25.359238 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:25.359219 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-must-gather-9qx4b/must-gather-v4shm" podUID="aefde989-67b4-44c1-816d-fa8ea22048bc" containerName="copy" containerID="cri-o://e6217852884a804af15234f023f97a77c7bc14a77731e829d9a527778c1bf349" gracePeriod=2 Jul 09 10:23:25.361342 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:25.361320 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-9qx4b/must-gather-v4shm"] Jul 09 10:23:25.361473 ip-10-0-135-204 kubenswrapper[2578]: E0709 10:23:25.361447 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-v4shm\" is forbidden: User \"system:node:ip-10-0-135-204.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-9qx4b\": no relationship found between node 'ip-10-0-135-204.ec2.internal' and this object" podUID="aefde989-67b4-44c1-816d-fa8ea22048bc" pod="openshift-must-gather-9qx4b/must-gather-v4shm" Jul 09 10:23:25.444454 ip-10-0-135-204 kubenswrapper[2578]: E0709 10:23:25.444418 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-v4shm\" is forbidden: User \"system:node:ip-10-0-135-204.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-9qx4b\": no relationship found between node 'ip-10-0-135-204.ec2.internal' and this object" podUID="aefde989-67b4-44c1-816d-fa8ea22048bc" pod="openshift-must-gather-9qx4b/must-gather-v4shm" Jul 09 10:23:25.594793 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:25.594771 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-9qx4b_must-gather-v4shm_aefde989-67b4-44c1-816d-fa8ea22048bc/copy/0.log" Jul 09 10:23:25.596322 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:25.596302 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9qx4b/must-gather-v4shm" Jul 09 10:23:25.646488 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:25.646467 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/aefde989-67b4-44c1-816d-fa8ea22048bc-must-gather-output\") pod \"aefde989-67b4-44c1-816d-fa8ea22048bc\" (UID: \"aefde989-67b4-44c1-816d-fa8ea22048bc\") " Jul 09 10:23:25.646579 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:25.646509 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gnzms\" (UniqueName: \"kubernetes.io/projected/aefde989-67b4-44c1-816d-fa8ea22048bc-kube-api-access-gnzms\") pod \"aefde989-67b4-44c1-816d-fa8ea22048bc\" (UID: \"aefde989-67b4-44c1-816d-fa8ea22048bc\") " Jul 09 10:23:25.647905 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:25.647882 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aefde989-67b4-44c1-816d-fa8ea22048bc-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "aefde989-67b4-44c1-816d-fa8ea22048bc" (UID: "aefde989-67b4-44c1-816d-fa8ea22048bc"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 10:23:25.648712 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:25.648681 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aefde989-67b4-44c1-816d-fa8ea22048bc-kube-api-access-gnzms" (OuterVolumeSpecName: "kube-api-access-gnzms") pod "aefde989-67b4-44c1-816d-fa8ea22048bc" (UID: "aefde989-67b4-44c1-816d-fa8ea22048bc"). InnerVolumeSpecName "kube-api-access-gnzms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 10:23:25.747537 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:25.747512 2578 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/aefde989-67b4-44c1-816d-fa8ea22048bc-must-gather-output\") on node \"ip-10-0-135-204.ec2.internal\" DevicePath \"\"" Jul 09 10:23:25.747537 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:25.747534 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnzms\" (UniqueName: \"kubernetes.io/projected/aefde989-67b4-44c1-816d-fa8ea22048bc-kube-api-access-gnzms\") on node \"ip-10-0-135-204.ec2.internal\" DevicePath \"\"" Jul 09 10:23:25.924590 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:25.924526 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-9qx4b_must-gather-v4shm_aefde989-67b4-44c1-816d-fa8ea22048bc/copy/0.log" Jul 09 10:23:25.924880 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:25.924828 2578 generic.go:356] "Generic (PLEG): container finished" podID="aefde989-67b4-44c1-816d-fa8ea22048bc" containerID="e6217852884a804af15234f023f97a77c7bc14a77731e829d9a527778c1bf349" exitCode=143 Jul 09 10:23:25.924996 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:25.924901 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9qx4b/must-gather-v4shm" Jul 09 10:23:25.924996 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:25.924962 2578 scope.go:117] "RemoveContainer" containerID="e6217852884a804af15234f023f97a77c7bc14a77731e829d9a527778c1bf349" Jul 09 10:23:25.933333 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:25.933305 2578 scope.go:117] "RemoveContainer" containerID="ab751f674480eb5fe1f84850d61fb0a3f59ea413d1648594ab21235a384e5e00" Jul 09 10:23:25.944940 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:25.944926 2578 scope.go:117] "RemoveContainer" containerID="e6217852884a804af15234f023f97a77c7bc14a77731e829d9a527778c1bf349" Jul 09 10:23:25.945168 ip-10-0-135-204 kubenswrapper[2578]: E0709 10:23:25.945153 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6217852884a804af15234f023f97a77c7bc14a77731e829d9a527778c1bf349\": container with ID starting with e6217852884a804af15234f023f97a77c7bc14a77731e829d9a527778c1bf349 not found: ID does not exist" containerID="e6217852884a804af15234f023f97a77c7bc14a77731e829d9a527778c1bf349" Jul 09 10:23:25.945206 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:25.945175 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6217852884a804af15234f023f97a77c7bc14a77731e829d9a527778c1bf349"} err="failed to get container status \"e6217852884a804af15234f023f97a77c7bc14a77731e829d9a527778c1bf349\": rpc error: code = NotFound desc = could not find container \"e6217852884a804af15234f023f97a77c7bc14a77731e829d9a527778c1bf349\": container with ID starting with e6217852884a804af15234f023f97a77c7bc14a77731e829d9a527778c1bf349 not found: ID does not exist" Jul 09 10:23:25.945206 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:25.945191 2578 scope.go:117] "RemoveContainer" containerID="ab751f674480eb5fe1f84850d61fb0a3f59ea413d1648594ab21235a384e5e00" Jul 09 10:23:25.945384 ip-10-0-135-204 kubenswrapper[2578]: E0709 10:23:25.945369 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab751f674480eb5fe1f84850d61fb0a3f59ea413d1648594ab21235a384e5e00\": container with ID starting with ab751f674480eb5fe1f84850d61fb0a3f59ea413d1648594ab21235a384e5e00 not found: ID does not exist" containerID="ab751f674480eb5fe1f84850d61fb0a3f59ea413d1648594ab21235a384e5e00" Jul 09 10:23:25.945419 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:25.945388 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab751f674480eb5fe1f84850d61fb0a3f59ea413d1648594ab21235a384e5e00"} err="failed to get container status \"ab751f674480eb5fe1f84850d61fb0a3f59ea413d1648594ab21235a384e5e00\": rpc error: code = NotFound desc = could not find container \"ab751f674480eb5fe1f84850d61fb0a3f59ea413d1648594ab21235a384e5e00\": container with ID starting with ab751f674480eb5fe1f84850d61fb0a3f59ea413d1648594ab21235a384e5e00 not found: ID does not exist" Jul 09 10:23:27.442639 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:27.442607 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aefde989-67b4-44c1-816d-fa8ea22048bc" path="/var/lib/kubelet/pods/aefde989-67b4-44c1-816d-fa8ea22048bc/volumes" Jul 09 10:23:28.051013 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:28.050978 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-q9lb8_e4b1e242-011d-4b72-8aaa-1e6d75842178/kube-state-metrics/0.log" Jul 09 10:23:28.087622 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:28.087601 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-q9lb8_e4b1e242-011d-4b72-8aaa-1e6d75842178/kube-rbac-proxy-main/0.log" Jul 09 10:23:28.117156 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:28.117125 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-q9lb8_e4b1e242-011d-4b72-8aaa-1e6d75842178/kube-rbac-proxy-self/0.log" Jul 09 10:23:28.211609 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:28.211584 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-hgks7_675a460e-eff5-4cc7-b339-d104c9bbe310/monitoring-plugin/0.log" Jul 09 10:23:28.332635 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:28.332568 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s2dl4_6b5a52bf-1368-4232-a345-09f786e28558/node-exporter/0.log" Jul 09 10:23:28.362144 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:28.362124 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s2dl4_6b5a52bf-1368-4232-a345-09f786e28558/kube-rbac-proxy/0.log" Jul 09 10:23:28.390928 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:28.390907 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s2dl4_6b5a52bf-1368-4232-a345-09f786e28558/init-textfile/0.log" Jul 09 10:23:28.675286 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:28.675256 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cb032b45-8976-4ea6-9b05-1dbb6e6ba94d/prometheus/0.log" Jul 09 10:23:28.709759 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:28.709736 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cb032b45-8976-4ea6-9b05-1dbb6e6ba94d/config-reloader/0.log" Jul 09 10:23:28.752138 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:28.752120 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cb032b45-8976-4ea6-9b05-1dbb6e6ba94d/thanos-sidecar/0.log" Jul 09 10:23:28.782332 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:28.782314 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cb032b45-8976-4ea6-9b05-1dbb6e6ba94d/kube-rbac-proxy-web/0.log" Jul 09 10:23:28.817348 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:28.817311 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cb032b45-8976-4ea6-9b05-1dbb6e6ba94d/kube-rbac-proxy/0.log" Jul 09 10:23:28.849930 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:28.849903 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cb032b45-8976-4ea6-9b05-1dbb6e6ba94d/kube-rbac-proxy-thanos/0.log" Jul 09 10:23:28.885661 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:28.885646 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cb032b45-8976-4ea6-9b05-1dbb6e6ba94d/init-config-reloader/0.log" Jul 09 10:23:28.953965 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:28.953894 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-dwqgj_1b5a080b-1c43-4958-899a-8fcabb524ba3/prometheus-operator/0.log" Jul 09 10:23:29.028826 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:29.028807 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-dwqgj_1b5a080b-1c43-4958-899a-8fcabb524ba3/kube-rbac-proxy/0.log" Jul 09 10:23:31.617928 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:31.617903 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-85f5448947-c6q9r_9f4cb38c-f6ba-4250-a710-a465a6d1d74c/console/0.log" Jul 09 10:23:31.650472 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:31.650440 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-685bfb5f64-9hk94_18bcc96c-0950-49b5-b109-fdd52cfd0616/download-server/0.log" Jul 09 10:23:32.837896 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:32.837867 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-q8qv5/perf-node-gather-daemonset-7jn7d"] Jul 09 10:23:32.838241 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:32.838191 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aefde989-67b4-44c1-816d-fa8ea22048bc" containerName="copy" Jul 09 10:23:32.838241 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:32.838202 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="aefde989-67b4-44c1-816d-fa8ea22048bc" containerName="copy" Jul 09 10:23:32.838241 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:32.838220 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aefde989-67b4-44c1-816d-fa8ea22048bc" containerName="gather" Jul 09 10:23:32.838241 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:32.838225 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="aefde989-67b4-44c1-816d-fa8ea22048bc" containerName="gather" Jul 09 10:23:32.838368 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:32.838273 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="aefde989-67b4-44c1-816d-fa8ea22048bc" containerName="gather" Jul 09 10:23:32.838368 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:32.838285 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="aefde989-67b4-44c1-816d-fa8ea22048bc" containerName="copy" Jul 09 10:23:32.841732 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:32.841714 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q8qv5/perf-node-gather-daemonset-7jn7d" Jul 09 10:23:32.844237 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:32.844219 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-q8qv5\"/\"default-dockercfg-4jsqr\"" Jul 09 10:23:32.844322 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:32.844219 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-q8qv5\"/\"kube-root-ca.crt\"" Jul 09 10:23:32.845090 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:32.845075 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-q8qv5\"/\"openshift-service-ca.crt\"" Jul 09 10:23:32.851226 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:32.851209 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q8qv5/perf-node-gather-daemonset-7jn7d"] Jul 09 10:23:32.903030 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:32.903005 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9jdx\" (UniqueName: \"kubernetes.io/projected/0c7f915f-9942-46fe-bea5-fcb8720dbd25-kube-api-access-x9jdx\") pod \"perf-node-gather-daemonset-7jn7d\" (UID: \"0c7f915f-9942-46fe-bea5-fcb8720dbd25\") " pod="openshift-must-gather-q8qv5/perf-node-gather-daemonset-7jn7d" Jul 09 10:23:32.903128 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:32.903047 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0c7f915f-9942-46fe-bea5-fcb8720dbd25-podres\") pod \"perf-node-gather-daemonset-7jn7d\" (UID: \"0c7f915f-9942-46fe-bea5-fcb8720dbd25\") " pod="openshift-must-gather-q8qv5/perf-node-gather-daemonset-7jn7d" Jul 09 10:23:32.903128 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:32.903085 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0c7f915f-9942-46fe-bea5-fcb8720dbd25-lib-modules\") pod \"perf-node-gather-daemonset-7jn7d\" (UID: \"0c7f915f-9942-46fe-bea5-fcb8720dbd25\") " pod="openshift-must-gather-q8qv5/perf-node-gather-daemonset-7jn7d" Jul 09 10:23:32.903203 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:32.903136 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0c7f915f-9942-46fe-bea5-fcb8720dbd25-proc\") pod \"perf-node-gather-daemonset-7jn7d\" (UID: \"0c7f915f-9942-46fe-bea5-fcb8720dbd25\") " pod="openshift-must-gather-q8qv5/perf-node-gather-daemonset-7jn7d" Jul 09 10:23:32.903203 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:32.903164 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0c7f915f-9942-46fe-bea5-fcb8720dbd25-sys\") pod \"perf-node-gather-daemonset-7jn7d\" (UID: \"0c7f915f-9942-46fe-bea5-fcb8720dbd25\") " pod="openshift-must-gather-q8qv5/perf-node-gather-daemonset-7jn7d" Jul 09 10:23:33.004184 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:33.004156 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0c7f915f-9942-46fe-bea5-fcb8720dbd25-lib-modules\") pod \"perf-node-gather-daemonset-7jn7d\" (UID: \"0c7f915f-9942-46fe-bea5-fcb8720dbd25\") " pod="openshift-must-gather-q8qv5/perf-node-gather-daemonset-7jn7d" Jul 09 10:23:33.004296 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:33.004188 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0c7f915f-9942-46fe-bea5-fcb8720dbd25-proc\") pod \"perf-node-gather-daemonset-7jn7d\" (UID: \"0c7f915f-9942-46fe-bea5-fcb8720dbd25\") " pod="openshift-must-gather-q8qv5/perf-node-gather-daemonset-7jn7d" Jul 09 10:23:33.004296 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:33.004208 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0c7f915f-9942-46fe-bea5-fcb8720dbd25-sys\") pod \"perf-node-gather-daemonset-7jn7d\" (UID: \"0c7f915f-9942-46fe-bea5-fcb8720dbd25\") " pod="openshift-must-gather-q8qv5/perf-node-gather-daemonset-7jn7d" Jul 09 10:23:33.004296 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:33.004243 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9jdx\" (UniqueName: \"kubernetes.io/projected/0c7f915f-9942-46fe-bea5-fcb8720dbd25-kube-api-access-x9jdx\") pod \"perf-node-gather-daemonset-7jn7d\" (UID: \"0c7f915f-9942-46fe-bea5-fcb8720dbd25\") " pod="openshift-must-gather-q8qv5/perf-node-gather-daemonset-7jn7d" Jul 09 10:23:33.004296 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:33.004281 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0c7f915f-9942-46fe-bea5-fcb8720dbd25-podres\") pod \"perf-node-gather-daemonset-7jn7d\" (UID: \"0c7f915f-9942-46fe-bea5-fcb8720dbd25\") " pod="openshift-must-gather-q8qv5/perf-node-gather-daemonset-7jn7d" Jul 09 10:23:33.004425 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:33.004303 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0c7f915f-9942-46fe-bea5-fcb8720dbd25-sys\") pod \"perf-node-gather-daemonset-7jn7d\" (UID: \"0c7f915f-9942-46fe-bea5-fcb8720dbd25\") " pod="openshift-must-gather-q8qv5/perf-node-gather-daemonset-7jn7d" Jul 09 10:23:33.004425 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:33.004303 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0c7f915f-9942-46fe-bea5-fcb8720dbd25-proc\") pod \"perf-node-gather-daemonset-7jn7d\" (UID: \"0c7f915f-9942-46fe-bea5-fcb8720dbd25\") " pod="openshift-must-gather-q8qv5/perf-node-gather-daemonset-7jn7d" Jul 09 10:23:33.004425 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:33.004341 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0c7f915f-9942-46fe-bea5-fcb8720dbd25-lib-modules\") pod \"perf-node-gather-daemonset-7jn7d\" (UID: \"0c7f915f-9942-46fe-bea5-fcb8720dbd25\") " pod="openshift-must-gather-q8qv5/perf-node-gather-daemonset-7jn7d" Jul 09 10:23:33.004425 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:33.004377 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0c7f915f-9942-46fe-bea5-fcb8720dbd25-podres\") pod \"perf-node-gather-daemonset-7jn7d\" (UID: \"0c7f915f-9942-46fe-bea5-fcb8720dbd25\") " pod="openshift-must-gather-q8qv5/perf-node-gather-daemonset-7jn7d" Jul 09 10:23:33.012519 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:33.012495 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9jdx\" (UniqueName: \"kubernetes.io/projected/0c7f915f-9942-46fe-bea5-fcb8720dbd25-kube-api-access-x9jdx\") pod \"perf-node-gather-daemonset-7jn7d\" (UID: \"0c7f915f-9942-46fe-bea5-fcb8720dbd25\") " pod="openshift-must-gather-q8qv5/perf-node-gather-daemonset-7jn7d" Jul 09 10:23:33.151902 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:33.151870 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q8qv5/perf-node-gather-daemonset-7jn7d" Jul 09 10:23:33.160122 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:33.160099 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ztphn_f5bf5aed-640a-478f-a004-94b74b6d3ed9/dns/0.log" Jul 09 10:23:33.193111 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:33.193088 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ztphn_f5bf5aed-640a-478f-a004-94b74b6d3ed9/kube-rbac-proxy/0.log" Jul 09 10:23:33.291025 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:33.290997 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q8qv5/perf-node-gather-daemonset-7jn7d"] Jul 09 10:23:33.294700 ip-10-0-135-204 kubenswrapper[2578]: W0709 10:23:33.294666 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0c7f915f_9942_46fe_bea5_fcb8720dbd25.slice/crio-0263129926fe18bccad216fc36ed4bd9e7b6d97ea3f36d90fc767a1d99892850 WatchSource:0}: Error finding container 0263129926fe18bccad216fc36ed4bd9e7b6d97ea3f36d90fc767a1d99892850: Status 404 returned error can't find the container with id 0263129926fe18bccad216fc36ed4bd9e7b6d97ea3f36d90fc767a1d99892850 Jul 09 10:23:33.322548 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:33.322530 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nx4l4_32307b23-721f-4ed0-add4-542ed80a29d8/dns-node-resolver/0.log" Jul 09 10:23:33.941978 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:33.941950 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qbtkh_f8587efa-7f0b-4a11-b03f-e08b942aa1f1/node-ca/0.log" Jul 09 10:23:33.957064 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:33.957039 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q8qv5/perf-node-gather-daemonset-7jn7d" event={"ID":"0c7f915f-9942-46fe-bea5-fcb8720dbd25","Type":"ContainerStarted","Data":"23c977a093bbc2b02aed19a271664c42285191b2c28a64f58161fa08690b9167"} Jul 09 10:23:33.957170 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:33.957080 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q8qv5/perf-node-gather-daemonset-7jn7d" event={"ID":"0c7f915f-9942-46fe-bea5-fcb8720dbd25","Type":"ContainerStarted","Data":"0263129926fe18bccad216fc36ed4bd9e7b6d97ea3f36d90fc767a1d99892850"} Jul 09 10:23:33.957170 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:33.957158 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-q8qv5/perf-node-gather-daemonset-7jn7d" Jul 09 10:23:35.209164 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:35.209135 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-cbnbv_6a11f547-4bad-4cc3-9a49-bc04f9eac0e9/serve-healthcheck-canary/0.log" Jul 09 10:23:35.696705 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:35.696675 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-456st_dffd25e2-f81c-4c5f-afeb-1b56c2930e68/kube-rbac-proxy/0.log" Jul 09 10:23:35.717893 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:35.717871 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-456st_dffd25e2-f81c-4c5f-afeb-1b56c2930e68/exporter/0.log" Jul 09 10:23:35.740601 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:35.740576 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-456st_dffd25e2-f81c-4c5f-afeb-1b56c2930e68/extractor/0.log" Jul 09 10:23:38.115150 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:38.115119 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-fd65d7d6-r9xks_65076f9d-75bb-4e52-96c5-36542a147781/server/0.log" Jul 09 10:23:38.303301 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:38.303271 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-rs6b4_0584ca51-2a1c-427c-a258-6947af590a68/s3-init/0.log" Jul 09 10:23:38.332331 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:38.332311 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-rs6b4_0584ca51-2a1c-427c-a258-6947af590a68/download-hf-model/0.log" Jul 09 10:23:38.380167 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:38.380144 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-7bqkz_e8001e12-b4b6-4781-8210-b4e0be833a53/seaweedfs/0.log" Jul 09 10:23:39.971078 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:39.971050 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-q8qv5/perf-node-gather-daemonset-7jn7d" Jul 09 10:23:39.993370 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:39.993327 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-q8qv5/perf-node-gather-daemonset-7jn7d" podStartSLOduration=7.993307923 podStartE2EDuration="7.993307923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 10:23:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:23:33.977722094 +0000 UTC m=+2999.069476014" watchObservedRunningTime="2026-07-09 10:23:39.993307923 +0000 UTC m=+3005.085061843" Jul 09 10:23:44.156117 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:44.156090 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rspgp_65d9652f-67de-405c-8e07-aa737e9205d8/kube-multus-additional-cni-plugins/0.log" Jul 09 10:23:44.185278 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:44.185242 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rspgp_65d9652f-67de-405c-8e07-aa737e9205d8/egress-router-binary-copy/0.log" Jul 09 10:23:44.208607 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:44.208573 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rspgp_65d9652f-67de-405c-8e07-aa737e9205d8/cni-plugins/0.log" Jul 09 10:23:44.235506 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:44.235478 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rspgp_65d9652f-67de-405c-8e07-aa737e9205d8/bond-cni-plugin/0.log" Jul 09 10:23:44.263500 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:44.263478 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rspgp_65d9652f-67de-405c-8e07-aa737e9205d8/routeoverride-cni/0.log" Jul 09 10:23:44.289799 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:44.289777 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rspgp_65d9652f-67de-405c-8e07-aa737e9205d8/whereabouts-cni-bincopy/0.log" Jul 09 10:23:44.318773 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:44.318750 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rspgp_65d9652f-67de-405c-8e07-aa737e9205d8/whereabouts-cni/0.log" Jul 09 10:23:44.352553 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:44.352536 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-clbg8_6e7da002-bfa6-4527-bfb9-ee3bc36677f1/kube-multus/0.log" Jul 09 10:23:44.528693 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:44.528631 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-wsdxs_c0652b01-6ddc-4820-8fbe-c2e5b643f663/network-metrics-daemon/0.log" Jul 09 10:23:44.545799 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:44.545775 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-wsdxs_c0652b01-6ddc-4820-8fbe-c2e5b643f663/kube-rbac-proxy/0.log" Jul 09 10:23:45.748467 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:45.748437 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xf4sw_1677420d-7376-45ad-ac67-20128aec5756/ovn-controller/0.log" Jul 09 10:23:45.769912 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:45.769881 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xf4sw_1677420d-7376-45ad-ac67-20128aec5756/ovn-acl-logging/1.log" Jul 09 10:23:45.770032 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:45.769913 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xf4sw_1677420d-7376-45ad-ac67-20128aec5756/ovn-acl-logging/0.log" Jul 09 10:23:45.785738 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:45.785720 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xf4sw_1677420d-7376-45ad-ac67-20128aec5756/kube-rbac-proxy-node/0.log" Jul 09 10:23:45.811029 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:45.811008 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xf4sw_1677420d-7376-45ad-ac67-20128aec5756/kube-rbac-proxy-ovn-metrics/0.log" Jul 09 10:23:45.835256 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:45.835236 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xf4sw_1677420d-7376-45ad-ac67-20128aec5756/northd/0.log" Jul 09 10:23:45.857688 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:45.857669 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xf4sw_1677420d-7376-45ad-ac67-20128aec5756/nbdb/0.log" Jul 09 10:23:45.879378 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:45.879356 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xf4sw_1677420d-7376-45ad-ac67-20128aec5756/sbdb/0.log" Jul 09 10:23:45.919920 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:45.919904 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xf4sw_1677420d-7376-45ad-ac67-20128aec5756/ovnkube-controller/0.log" Jul 09 10:23:47.007521 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:47.007492 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-gqlhc_53e2e8a5-34d5-4317-9733-143a53880b9b/network-check-target-container/0.log" Jul 09 10:23:48.038724 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:48.038697 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-r6npq_c7314861-eada-458d-bf61-21da88f8b1c8/iptables-alerter/0.log" Jul 09 10:23:48.784451 ip-10-0-135-204 kubenswrapper[2578]: I0709 10:23:48.784376 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-dnzdl_46a45a21-2ddc-4b0b-9988-911cde1598f7/tuned/0.log"