Jul 21 02:07:19.155419 ip-10-0-141-37 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 21 02:07:19.155429 ip-10-0-141-37 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 21 02:07:19.155437 ip-10-0-141-37 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 21 02:07:19.155675 ip-10-0-141-37 systemd[1]: Failed to start Kubernetes Kubelet. Jul 21 02:07:29.306697 ip-10-0-141-37 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 21 02:07:29.306717 ip-10-0-141-37 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 0fcbb346802449c4b1d8aec48802de41 -- Jul 21 02:09:25.857577 ip-10-0-141-37 systemd[1]: Starting Kubernetes Kubelet... Jul 21 02:09:28.192264 ip-10-0-141-37 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 21 02:09:28.192264 ip-10-0-141-37 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 21 02:09:28.192264 ip-10-0-141-37 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 21 02:09:28.192264 ip-10-0-141-37 kubenswrapper[2574]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 21 02:09:28.192264 ip-10-0-141-37 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 21 02:09:28.194563 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.194397 2574 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 21 02:09:28.198817 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.198798 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 02:09:28.198817 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.198816 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 02:09:28.198888 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.198820 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 02:09:28.198888 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.198825 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 02:09:28.198888 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.198828 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 02:09:28.198888 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.198832 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 02:09:28.198888 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.198835 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 02:09:28.198888 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.198838 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 02:09:28.198888 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.198842 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 02:09:28.198888 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.198845 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 02:09:28.198888 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.198848 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 02:09:28.198888 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.198851 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 02:09:28.198888 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.198854 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 02:09:28.198888 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.198857 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 02:09:28.198888 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.198860 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 02:09:28.198888 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.198863 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 02:09:28.198888 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.198866 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 02:09:28.198888 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.198869 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 02:09:28.198888 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.198872 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 02:09:28.198888 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.198881 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 02:09:28.198888 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.198884 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 02:09:28.198888 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.198887 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 02:09:28.199399 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.198890 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 02:09:28.199399 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.198893 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 21 02:09:28.199399 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.198896 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 02:09:28.199399 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.198899 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 02:09:28.199399 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.198902 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 02:09:28.199399 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.198905 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 02:09:28.199399 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.198908 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 02:09:28.199399 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.198911 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 02:09:28.199399 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.198914 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 02:09:28.199399 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.198917 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 02:09:28.199399 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.198920 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 02:09:28.199399 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.198922 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 02:09:28.199399 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.198925 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 02:09:28.199399 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.198928 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 02:09:28.199399 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.198930 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 02:09:28.199399 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.198933 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 02:09:28.199399 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.198936 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 02:09:28.199399 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.198939 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 02:09:28.199399 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.198941 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 02:09:28.199399 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.198944 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 02:09:28.199928 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.198946 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 02:09:28.199928 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.198949 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 02:09:28.199928 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.198952 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 02:09:28.199928 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.198955 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 02:09:28.199928 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.198958 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 02:09:28.199928 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.198961 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 02:09:28.199928 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.198964 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 02:09:28.199928 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.198967 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 02:09:28.199928 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.198969 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 02:09:28.199928 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.198972 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 02:09:28.199928 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.198975 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 02:09:28.199928 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.198977 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 02:09:28.199928 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.198980 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 02:09:28.199928 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.198983 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 02:09:28.199928 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.198986 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 02:09:28.199928 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.198989 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 02:09:28.199928 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.198992 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 02:09:28.199928 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.198994 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 02:09:28.199928 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.198997 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 02:09:28.199928 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.199000 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 02:09:28.200446 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.199003 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 02:09:28.200446 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.199005 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 02:09:28.200446 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.199008 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 02:09:28.200446 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.199011 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 02:09:28.200446 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.199013 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 02:09:28.200446 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.199016 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 02:09:28.200446 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.199018 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 02:09:28.200446 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.199023 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 02:09:28.200446 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.199028 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 02:09:28.200446 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.199030 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 02:09:28.200446 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.199033 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 02:09:28.200446 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.199036 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 02:09:28.200446 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.199039 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 02:09:28.200446 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.199042 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 02:09:28.200446 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.199045 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 02:09:28.200446 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.199048 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 02:09:28.200446 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.199051 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 02:09:28.200446 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.199054 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 02:09:28.200446 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.199061 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 02:09:28.200446 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.199064 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 02:09:28.200956 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.199066 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 02:09:28.200956 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.199071 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 02:09:28.200956 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.199074 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 02:09:28.200956 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.199078 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 02:09:28.200956 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.199081 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 02:09:28.200956 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.199084 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 02:09:28.200956 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.199087 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 02:09:28.200956 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.199090 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 02:09:28.200956 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.199093 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 02:09:28.200956 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.199096 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 02:09:28.200956 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.199099 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 02:09:28.200956 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.199101 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 02:09:28.200956 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.199104 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 02:09:28.200956 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.199109 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 02:09:28.200956 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.199112 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 02:09:28.200956 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.199116 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 02:09:28.200956 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.199119 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 02:09:28.200956 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.199945 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 02:09:28.200956 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.199953 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 02:09:28.201480 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.199956 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 02:09:28.201480 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.199963 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 02:09:28.201480 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.199967 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 02:09:28.201480 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.199970 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 02:09:28.201480 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.199974 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 02:09:28.201480 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.199977 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 02:09:28.201480 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.199980 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 02:09:28.201480 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.199983 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 02:09:28.201480 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.199985 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 02:09:28.201480 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.199988 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 02:09:28.201480 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.199993 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 02:09:28.201480 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.199997 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 02:09:28.201480 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200001 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 02:09:28.201480 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200004 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 02:09:28.201480 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200007 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 02:09:28.201480 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200010 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 02:09:28.201480 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200015 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 02:09:28.201480 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200018 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 02:09:28.201480 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200022 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 02:09:28.202002 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200026 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 02:09:28.202002 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200029 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 02:09:28.202002 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200032 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 02:09:28.202002 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200035 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 02:09:28.202002 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200038 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 02:09:28.202002 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200040 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 02:09:28.202002 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200043 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 02:09:28.202002 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200046 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 02:09:28.202002 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200049 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 02:09:28.202002 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200052 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 02:09:28.202002 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200055 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 02:09:28.202002 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200057 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 02:09:28.202002 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200060 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 02:09:28.202002 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200063 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 02:09:28.202002 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200065 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 02:09:28.202002 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200075 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 02:09:28.202002 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200078 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 21 02:09:28.202002 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200080 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 02:09:28.202002 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200084 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 02:09:28.202002 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200088 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 02:09:28.202525 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200090 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 02:09:28.202525 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200094 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 02:09:28.202525 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200098 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 02:09:28.202525 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200101 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 02:09:28.202525 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200104 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 02:09:28.202525 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200106 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 02:09:28.202525 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200109 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 02:09:28.202525 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200112 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 02:09:28.202525 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200115 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 02:09:28.202525 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200118 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 02:09:28.202525 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200121 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 02:09:28.202525 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200124 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 02:09:28.202525 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200127 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 02:09:28.202525 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200129 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 02:09:28.202525 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200132 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 02:09:28.202525 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200135 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 02:09:28.202525 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200137 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 02:09:28.202525 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200140 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 02:09:28.202525 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200143 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 02:09:28.202525 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200145 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 02:09:28.203038 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200148 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 02:09:28.203038 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200151 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 02:09:28.203038 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200153 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 02:09:28.203038 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200156 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 02:09:28.203038 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200159 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 02:09:28.203038 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200161 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 02:09:28.203038 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200164 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 02:09:28.203038 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200166 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 02:09:28.203038 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200175 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 02:09:28.203038 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200178 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 02:09:28.203038 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200181 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 02:09:28.203038 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200184 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 02:09:28.203038 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200187 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 02:09:28.203038 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200190 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 02:09:28.203038 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200193 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 02:09:28.203038 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200196 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 02:09:28.203038 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200200 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 02:09:28.203038 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200203 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 02:09:28.203038 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200206 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 02:09:28.203038 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200209 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 02:09:28.203562 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200211 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 02:09:28.203562 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200214 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 02:09:28.203562 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200216 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 02:09:28.203562 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200219 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 02:09:28.203562 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200222 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 02:09:28.203562 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200224 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 02:09:28.203562 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200227 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 02:09:28.203562 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200230 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 02:09:28.203562 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200232 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 02:09:28.203562 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200235 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 02:09:28.203562 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200238 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 02:09:28.203562 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200240 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 02:09:28.203562 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200243 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 02:09:28.203562 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200246 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 02:09:28.203562 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200248 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 02:09:28.203562 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200251 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 02:09:28.203562 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200253 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 02:09:28.203562 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200256 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 02:09:28.203562 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200700 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 02:09:28.203562 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200706 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 02:09:28.204069 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200717 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 02:09:28.204069 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200721 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 02:09:28.204069 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200725 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 02:09:28.204069 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200728 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 02:09:28.204069 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200732 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 02:09:28.204069 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200735 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 02:09:28.204069 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200739 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 02:09:28.204069 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200742 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 02:09:28.204069 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200746 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 02:09:28.204069 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200748 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 02:09:28.204069 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200751 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 02:09:28.204069 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200754 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 02:09:28.204069 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200757 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 02:09:28.204069 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200760 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 02:09:28.204069 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200763 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 02:09:28.204069 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200766 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 02:09:28.204069 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200768 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 02:09:28.204069 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200771 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 02:09:28.204069 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200774 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 02:09:28.204558 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200777 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 02:09:28.204558 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200780 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 02:09:28.204558 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200782 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 02:09:28.204558 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200785 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 02:09:28.204558 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200788 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 02:09:28.204558 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200791 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 02:09:28.204558 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200794 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 02:09:28.204558 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200797 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 02:09:28.204558 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200800 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 02:09:28.204558 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200802 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 02:09:28.204558 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200808 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 02:09:28.204558 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200811 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 02:09:28.204558 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200814 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 02:09:28.204558 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200816 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 02:09:28.204558 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200825 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 02:09:28.204558 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200829 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 02:09:28.204558 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200832 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 02:09:28.204558 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200835 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 02:09:28.204558 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200837 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 02:09:28.205038 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200840 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 02:09:28.205038 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200843 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 02:09:28.205038 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200846 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 21 02:09:28.205038 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200848 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 02:09:28.205038 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200851 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 02:09:28.205038 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200855 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 02:09:28.205038 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200858 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 02:09:28.205038 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200861 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 02:09:28.205038 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200864 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 02:09:28.205038 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200867 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 02:09:28.205038 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200870 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 02:09:28.205038 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200873 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 02:09:28.205038 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200875 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 02:09:28.205038 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200878 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 02:09:28.205038 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200883 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 02:09:28.205038 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200887 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 02:09:28.205038 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200890 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 02:09:28.205038 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200893 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 02:09:28.205038 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200896 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 02:09:28.205038 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200899 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 02:09:28.205557 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200902 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 02:09:28.205557 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200905 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 02:09:28.205557 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200908 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 02:09:28.205557 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200912 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 02:09:28.205557 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200915 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 02:09:28.205557 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200918 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 02:09:28.205557 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200921 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 02:09:28.205557 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200923 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 02:09:28.205557 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200927 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 02:09:28.205557 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200930 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 02:09:28.205557 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200932 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 02:09:28.205557 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200935 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 02:09:28.205557 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200938 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 02:09:28.205557 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200940 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 02:09:28.205557 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200943 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 02:09:28.205557 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200946 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 02:09:28.205557 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200949 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 02:09:28.205557 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200951 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 02:09:28.205557 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200954 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 02:09:28.205557 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200957 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 02:09:28.206066 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200960 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 02:09:28.206066 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200963 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 02:09:28.206066 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200965 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 02:09:28.206066 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200968 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 02:09:28.206066 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200971 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 02:09:28.206066 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200974 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 02:09:28.206066 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200976 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 02:09:28.206066 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200979 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 02:09:28.206066 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200982 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 02:09:28.206066 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200984 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 02:09:28.206066 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200987 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 02:09:28.206066 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200991 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 02:09:28.206066 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200994 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 02:09:28.206066 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200997 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 02:09:28.206066 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.200999 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 02:09:28.206066 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.201003 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 02:09:28.206066 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.201005 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 02:09:28.206066 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.201008 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 02:09:28.206066 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.201011 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 02:09:28.206066 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201092 2574 flags.go:64] FLAG: --address="0.0.0.0" Jul 21 02:09:28.206066 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201100 2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 21 02:09:28.206606 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201114 2574 flags.go:64] FLAG: --anonymous-auth="true" Jul 21 02:09:28.206606 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201121 2574 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 21 02:09:28.206606 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201128 2574 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 21 02:09:28.206606 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201133 2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 21 02:09:28.206606 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201138 2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 21 02:09:28.206606 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201144 2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 21 02:09:28.206606 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201147 2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 21 02:09:28.206606 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201150 2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 21 02:09:28.206606 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201154 2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 21 02:09:28.206606 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201158 2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 21 02:09:28.206606 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201161 2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 21 02:09:28.206606 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201164 2574 flags.go:64] FLAG: --cgroup-root="" Jul 21 02:09:28.206606 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201167 2574 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 21 02:09:28.206606 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201170 2574 flags.go:64] FLAG: --client-ca-file="" Jul 21 02:09:28.206606 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201174 2574 flags.go:64] FLAG: --cloud-provider="external" Jul 21 02:09:28.206606 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201177 2574 flags.go:64] FLAG: --cluster-dns="[]" Jul 21 02:09:28.206606 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201181 2574 flags.go:64] FLAG: --cluster-domain="" Jul 21 02:09:28.206606 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201184 2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 21 02:09:28.206606 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201188 2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 21 02:09:28.206606 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201191 2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 21 02:09:28.206606 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201194 2574 flags.go:64] FLAG: --container-log-max-files="5" Jul 21 02:09:28.206606 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201198 2574 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 21 02:09:28.206606 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201205 2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 21 02:09:28.207190 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201209 2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 21 02:09:28.207190 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201212 2574 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 21 02:09:28.207190 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201215 2574 flags.go:64] FLAG: --contention-profiling="false" Jul 21 02:09:28.207190 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201220 2574 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 21 02:09:28.207190 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201223 2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 21 02:09:28.207190 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201227 2574 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 21 02:09:28.207190 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201230 2574 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 21 02:09:28.207190 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201234 2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 21 02:09:28.207190 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201238 2574 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 21 02:09:28.207190 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201241 2574 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 21 02:09:28.207190 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201244 2574 flags.go:64] FLAG: --enable-load-reader="false" Jul 21 02:09:28.207190 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201248 2574 flags.go:64] FLAG: --enable-server="true" Jul 21 02:09:28.207190 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201251 2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 21 02:09:28.207190 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201256 2574 flags.go:64] FLAG: --event-burst="100" Jul 21 02:09:28.207190 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201259 2574 flags.go:64] FLAG: --event-qps="50" Jul 21 02:09:28.207190 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201263 2574 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 21 02:09:28.207190 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201266 2574 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 21 02:09:28.207190 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201269 2574 flags.go:64] FLAG: --eviction-hard="" Jul 21 02:09:28.207190 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201273 2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 21 02:09:28.207190 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201278 2574 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 21 02:09:28.207190 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201283 2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 21 02:09:28.207190 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201288 2574 flags.go:64] FLAG: --eviction-soft="" Jul 21 02:09:28.207190 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201292 2574 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 21 02:09:28.207190 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201295 2574 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 21 02:09:28.207190 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201299 2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 21 02:09:28.207912 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201302 2574 flags.go:64] FLAG: --experimental-mounter-path="" Jul 21 02:09:28.207912 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201305 2574 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 21 02:09:28.207912 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201308 2574 flags.go:64] FLAG: --fail-swap-on="true" Jul 21 02:09:28.207912 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201312 2574 flags.go:64] FLAG: --feature-gates="" Jul 21 02:09:28.207912 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201316 2574 flags.go:64] FLAG: --file-check-frequency="20s" Jul 21 02:09:28.207912 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201320 2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 21 02:09:28.207912 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201325 2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 21 02:09:28.207912 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201329 2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 21 02:09:28.207912 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201332 2574 flags.go:64] FLAG: --healthz-port="10248" Jul 21 02:09:28.207912 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201335 2574 flags.go:64] FLAG: --help="false" Jul 21 02:09:28.207912 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201351 2574 flags.go:64] FLAG: --hostname-override="ip-10-0-141-37.ec2.internal" Jul 21 02:09:28.207912 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201355 2574 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 21 02:09:28.207912 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201358 2574 flags.go:64] FLAG: --http-check-frequency="20s" Jul 21 02:09:28.207912 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201362 2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 21 02:09:28.207912 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201366 2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 21 02:09:28.207912 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201370 2574 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 21 02:09:28.207912 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201373 2574 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 21 02:09:28.207912 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201376 2574 flags.go:64] FLAG: --image-service-endpoint="" Jul 21 02:09:28.207912 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201379 2574 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 21 02:09:28.207912 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201382 2574 flags.go:64] FLAG: --kube-api-burst="100" Jul 21 02:09:28.207912 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201386 2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 21 02:09:28.207912 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201389 2574 flags.go:64] FLAG: --kube-api-qps="50" Jul 21 02:09:28.207912 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201392 2574 flags.go:64] FLAG: --kube-reserved="" Jul 21 02:09:28.207912 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201395 2574 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 21 02:09:28.208559 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201398 2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 21 02:09:28.208559 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201401 2574 flags.go:64] FLAG: --kubelet-cgroups="" Jul 21 02:09:28.208559 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201404 2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 21 02:09:28.208559 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201407 2574 flags.go:64] FLAG: --lock-file="" Jul 21 02:09:28.208559 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201410 2574 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 21 02:09:28.208559 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201414 2574 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 21 02:09:28.208559 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201417 2574 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 21 02:09:28.208559 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201428 2574 flags.go:64] FLAG: --log-json-split-stream="false" Jul 21 02:09:28.208559 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201431 2574 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 21 02:09:28.208559 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201434 2574 flags.go:64] FLAG: --log-text-split-stream="false" Jul 21 02:09:28.208559 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201437 2574 flags.go:64] FLAG: --logging-format="text" Jul 21 02:09:28.208559 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201440 2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 21 02:09:28.208559 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201443 2574 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 21 02:09:28.208559 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201446 2574 flags.go:64] FLAG: --manifest-url="" Jul 21 02:09:28.208559 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201451 2574 flags.go:64] FLAG: --manifest-url-header="" Jul 21 02:09:28.208559 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201456 2574 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 21 02:09:28.208559 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201460 2574 flags.go:64] FLAG: --max-open-files="1000000" Jul 21 02:09:28.208559 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201465 2574 flags.go:64] FLAG: --max-pods="110" Jul 21 02:09:28.208559 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201468 2574 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 21 02:09:28.208559 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201471 2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 21 02:09:28.208559 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201474 2574 flags.go:64] FLAG: --memory-manager-policy="None" Jul 21 02:09:28.208559 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201477 2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 21 02:09:28.208559 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201481 2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 21 02:09:28.208559 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201485 2574 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 21 02:09:28.208559 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201488 2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 21 02:09:28.209221 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201495 2574 flags.go:64] FLAG: --node-status-max-images="50" Jul 21 02:09:28.209221 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201499 2574 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 21 02:09:28.209221 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201502 2574 flags.go:64] FLAG: --oom-score-adj="-999" Jul 21 02:09:28.209221 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201506 2574 flags.go:64] FLAG: --pod-cidr="" Jul 21 02:09:28.209221 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201509 2574 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 21 02:09:28.209221 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201516 2574 flags.go:64] FLAG: --pod-manifest-path="" Jul 21 02:09:28.209221 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201519 2574 flags.go:64] FLAG: --pod-max-pids="-1" Jul 21 02:09:28.209221 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201522 2574 flags.go:64] FLAG: --pods-per-core="0" Jul 21 02:09:28.209221 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201525 2574 flags.go:64] FLAG: --port="10250" Jul 21 02:09:28.209221 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201529 2574 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 21 02:09:28.209221 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201532 2574 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0444825e9e7c0e478" Jul 21 02:09:28.209221 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201535 2574 flags.go:64] FLAG: --qos-reserved="" Jul 21 02:09:28.209221 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201538 2574 flags.go:64] FLAG: --read-only-port="10255" Jul 21 02:09:28.209221 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201541 2574 flags.go:64] FLAG: --register-node="true" Jul 21 02:09:28.209221 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201545 2574 flags.go:64] FLAG: --register-with-taints="" Jul 21 02:09:28.209221 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201553 2574 flags.go:64] FLAG: --registry-burst="10" Jul 21 02:09:28.209221 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201556 2574 flags.go:64] FLAG: --registry-qps="5" Jul 21 02:09:28.209221 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201559 2574 flags.go:64] FLAG: --reserved-cpus="" Jul 21 02:09:28.209221 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201562 2574 flags.go:64] FLAG: --reserved-memory="" Jul 21 02:09:28.209221 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201566 2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 21 02:09:28.209221 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201570 2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 21 02:09:28.209221 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201573 2574 flags.go:64] FLAG: --rotate-certificates="false" Jul 21 02:09:28.209221 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201577 2574 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 21 02:09:28.209221 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201581 2574 flags.go:64] FLAG: --runonce="false" Jul 21 02:09:28.209221 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201584 2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 21 02:09:28.210181 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201588 2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 21 02:09:28.210181 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201592 2574 flags.go:64] FLAG: --seccomp-default="false" Jul 21 02:09:28.210181 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201595 2574 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 21 02:09:28.210181 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201598 2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 21 02:09:28.210181 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201601 2574 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 21 02:09:28.210181 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201604 2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 21 02:09:28.210181 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201608 2574 flags.go:64] FLAG: --storage-driver-password="root" Jul 21 02:09:28.210181 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201611 2574 flags.go:64] FLAG: --storage-driver-secure="false" Jul 21 02:09:28.210181 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201614 2574 flags.go:64] FLAG: --storage-driver-table="stats" Jul 21 02:09:28.210181 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201617 2574 flags.go:64] FLAG: --storage-driver-user="root" Jul 21 02:09:28.210181 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201627 2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 21 02:09:28.210181 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201631 2574 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 21 02:09:28.210181 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201634 2574 flags.go:64] FLAG: --system-cgroups="" Jul 21 02:09:28.210181 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201637 2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 21 02:09:28.210181 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201643 2574 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 21 02:09:28.210181 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201645 2574 flags.go:64] FLAG: --tls-cert-file="" Jul 21 02:09:28.210181 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201649 2574 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 21 02:09:28.210181 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201653 2574 flags.go:64] FLAG: --tls-min-version="" Jul 21 02:09:28.210181 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201656 2574 flags.go:64] FLAG: --tls-private-key-file="" Jul 21 02:09:28.210181 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201659 2574 flags.go:64] FLAG: --topology-manager-policy="none" Jul 21 02:09:28.210181 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201662 2574 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 21 02:09:28.210181 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201665 2574 flags.go:64] FLAG: --topology-manager-scope="container" Jul 21 02:09:28.210181 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201669 2574 flags.go:64] FLAG: --v="2" Jul 21 02:09:28.210181 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201673 2574 flags.go:64] FLAG: --version="false" Jul 21 02:09:28.210181 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201677 2574 flags.go:64] FLAG: --vmodule="" Jul 21 02:09:28.210944 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201681 2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 21 02:09:28.210944 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.201685 2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 21 02:09:28.210944 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.201852 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 02:09:28.210944 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.201856 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 02:09:28.210944 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.201861 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 02:09:28.210944 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.201864 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 02:09:28.210944 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.201868 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 02:09:28.210944 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.201872 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 02:09:28.210944 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.201875 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 02:09:28.210944 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.201878 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 02:09:28.210944 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.201881 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 02:09:28.210944 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.201884 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 02:09:28.210944 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.201887 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 02:09:28.210944 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.201890 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 02:09:28.210944 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.201893 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 02:09:28.210944 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.201896 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 02:09:28.210944 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.201899 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 02:09:28.210944 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.201902 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 02:09:28.210944 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.201912 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 02:09:28.210944 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.201915 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 02:09:28.211571 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.201918 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 02:09:28.211571 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.201921 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 02:09:28.211571 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.201923 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 02:09:28.211571 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.201926 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 02:09:28.211571 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.201929 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 02:09:28.211571 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.201934 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 02:09:28.211571 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.201937 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 02:09:28.211571 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.201941 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 02:09:28.211571 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.201944 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 02:09:28.211571 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.201947 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 02:09:28.211571 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.201950 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 02:09:28.211571 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.201953 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 02:09:28.211571 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.201956 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 02:09:28.211571 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.201959 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 02:09:28.211571 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.201961 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 02:09:28.211571 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.201964 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 02:09:28.211571 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.201969 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 02:09:28.211571 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.201971 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 02:09:28.211571 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.201974 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 02:09:28.212094 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.201979 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 02:09:28.212094 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.201981 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 02:09:28.212094 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.201984 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 02:09:28.212094 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.201987 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 02:09:28.212094 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.201990 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 02:09:28.212094 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.201992 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 02:09:28.212094 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.201997 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 02:09:28.212094 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.202000 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 02:09:28.212094 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.202003 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 02:09:28.212094 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.202007 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 02:09:28.212094 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.202010 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 02:09:28.212094 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.202013 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 02:09:28.212094 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.202017 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 02:09:28.212094 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.202020 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 02:09:28.212094 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.202023 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 02:09:28.212094 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.202027 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 02:09:28.212094 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.202031 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 02:09:28.212094 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.202034 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 02:09:28.212094 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.202038 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 02:09:28.212601 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.202040 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 02:09:28.212601 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.202043 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 02:09:28.212601 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.202047 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 02:09:28.212601 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.202050 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 02:09:28.212601 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.202053 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 02:09:28.212601 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.202056 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 02:09:28.212601 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.202059 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 02:09:28.212601 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.202061 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 02:09:28.212601 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.202065 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 02:09:28.212601 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.202067 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 02:09:28.212601 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.202072 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 02:09:28.212601 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.202075 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 02:09:28.212601 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.202078 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 02:09:28.212601 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.202083 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 02:09:28.212601 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.202085 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 02:09:28.212601 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.202088 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 02:09:28.212601 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.202091 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 02:09:28.212601 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.202094 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 21 02:09:28.212601 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.202097 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 02:09:28.212601 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.202099 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 02:09:28.213119 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.202102 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 02:09:28.213119 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.202105 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 02:09:28.213119 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.202108 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 02:09:28.213119 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.202111 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 02:09:28.213119 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.202114 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 02:09:28.213119 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.202117 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 02:09:28.213119 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.202120 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 02:09:28.213119 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.202123 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 02:09:28.213119 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.202126 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 02:09:28.213119 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.202129 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 02:09:28.213119 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.202131 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 02:09:28.213119 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.202134 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 02:09:28.213119 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.202137 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 02:09:28.213119 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.202140 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 02:09:28.213119 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.202143 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 02:09:28.213119 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.202145 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 02:09:28.213119 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.202148 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 02:09:28.213119 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.202150 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 02:09:28.213119 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.202153 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 02:09:28.213119 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.202155 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 02:09:28.213759 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.202158 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 02:09:28.213759 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.202161 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 02:09:28.213759 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.202165 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 02:09:28.213759 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.202946 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 21 02:09:28.213759 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.212499 2574 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 21 02:09:28.213759 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.212518 2574 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 21 02:09:28.213759 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.213555 2574 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 21 02:09:28.213759 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.213569 2574 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 21 02:09:28.213759 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213700 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 02:09:28.213759 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213707 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 02:09:28.213759 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213711 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 02:09:28.213759 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213716 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 02:09:28.213759 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213724 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 02:09:28.213759 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213729 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 02:09:28.214125 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213732 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 02:09:28.214125 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213736 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 02:09:28.214125 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213739 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 02:09:28.214125 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213742 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 02:09:28.214125 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213746 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 02:09:28.214125 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213749 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 02:09:28.214125 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213751 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 02:09:28.214125 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213755 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 02:09:28.214125 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213758 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 02:09:28.214125 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213761 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 02:09:28.214125 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213764 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 02:09:28.214125 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213770 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 02:09:28.214125 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213774 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 02:09:28.214125 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213776 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 02:09:28.214125 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213779 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 02:09:28.214125 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213783 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 02:09:28.214125 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213786 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 02:09:28.214125 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213788 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 02:09:28.214125 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213791 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 02:09:28.214125 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213794 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 02:09:28.214125 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213798 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 02:09:28.214674 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213800 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 02:09:28.214674 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213803 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 02:09:28.214674 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213813 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 02:09:28.214674 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213819 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 02:09:28.214674 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213822 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 02:09:28.214674 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213825 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 02:09:28.214674 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213828 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 02:09:28.214674 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213831 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 02:09:28.214674 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213834 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 02:09:28.214674 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213837 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 02:09:28.214674 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213839 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 02:09:28.214674 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213842 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 02:09:28.214674 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213846 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 02:09:28.214674 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213849 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 02:09:28.214674 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213852 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 02:09:28.214674 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213858 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 02:09:28.214674 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213861 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 02:09:28.214674 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213864 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 02:09:28.214674 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213867 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 02:09:28.214674 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213870 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 02:09:28.215186 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213873 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 02:09:28.215186 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213875 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 02:09:28.215186 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213878 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 02:09:28.215186 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213881 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 02:09:28.215186 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213886 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 02:09:28.215186 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213890 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 02:09:28.215186 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213893 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 02:09:28.215186 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213899 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 02:09:28.215186 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213902 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 02:09:28.215186 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213905 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 02:09:28.215186 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213908 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 02:09:28.215186 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213911 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 02:09:28.215186 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213914 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 02:09:28.215186 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213917 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 02:09:28.215186 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213920 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 02:09:28.215186 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213929 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 02:09:28.215186 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213932 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 02:09:28.215186 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213935 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 02:09:28.215186 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213938 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 02:09:28.215700 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213941 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 02:09:28.215700 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213946 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 02:09:28.215700 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213955 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 02:09:28.215700 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213958 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 02:09:28.215700 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213961 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 02:09:28.215700 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213964 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 02:09:28.215700 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213967 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 02:09:28.215700 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213970 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 02:09:28.215700 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213973 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 02:09:28.215700 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213976 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 02:09:28.215700 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213979 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 02:09:28.215700 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213982 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 02:09:28.215700 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213985 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 02:09:28.215700 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213988 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 02:09:28.215700 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213993 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 02:09:28.215700 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213996 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 02:09:28.215700 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.213999 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 02:09:28.215700 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214001 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 02:09:28.215700 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214004 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 02:09:28.216192 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214007 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 02:09:28.216192 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214010 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 02:09:28.216192 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214013 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 02:09:28.216192 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214015 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 02:09:28.216192 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214018 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 02:09:28.216192 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214021 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 02:09:28.216192 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214024 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 02:09:28.216192 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214028 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 02:09:28.216192 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214033 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 02:09:28.216192 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214036 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 02:09:28.216192 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214045 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 21 02:09:28.216192 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214050 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 02:09:28.216192 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214054 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 02:09:28.216192 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214058 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 02:09:28.216192 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.214064 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 21 02:09:28.216619 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214361 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 02:09:28.216619 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214371 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 02:09:28.216619 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214374 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 02:09:28.216619 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214378 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 02:09:28.216619 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214381 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 02:09:28.216619 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214385 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 02:09:28.216619 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214390 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 02:09:28.216619 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214393 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 02:09:28.216619 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214396 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 02:09:28.216619 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214399 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 02:09:28.216619 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214402 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 02:09:28.216619 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214405 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 02:09:28.216619 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214408 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 02:09:28.216619 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214414 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 21 02:09:28.216619 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214417 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 02:09:28.216619 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214419 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 02:09:28.216619 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214422 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 02:09:28.216619 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214425 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 02:09:28.216619 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214428 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 02:09:28.216619 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214431 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 02:09:28.217108 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214434 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 02:09:28.217108 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214437 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 02:09:28.217108 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214440 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 02:09:28.217108 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214443 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 02:09:28.217108 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214446 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 02:09:28.217108 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214448 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 02:09:28.217108 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214454 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 02:09:28.217108 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214457 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 02:09:28.217108 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214470 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 02:09:28.217108 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214473 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 02:09:28.217108 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214476 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 02:09:28.217108 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214479 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 02:09:28.217108 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214481 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 02:09:28.217108 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214484 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 02:09:28.217108 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214487 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 02:09:28.217108 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214492 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 02:09:28.217108 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214496 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 02:09:28.217108 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214499 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 02:09:28.217108 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214505 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 02:09:28.217603 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214508 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 02:09:28.217603 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214512 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 02:09:28.217603 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214514 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 02:09:28.217603 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214519 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 02:09:28.217603 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214522 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 02:09:28.217603 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214525 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 02:09:28.217603 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214529 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 02:09:28.217603 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214532 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 02:09:28.217603 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214535 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 02:09:28.217603 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214538 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 02:09:28.217603 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214541 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 02:09:28.217603 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214544 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 02:09:28.217603 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214550 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 02:09:28.217603 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214553 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 02:09:28.217603 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214556 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 02:09:28.217603 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214559 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 02:09:28.217603 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214562 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 02:09:28.217603 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214565 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 02:09:28.217603 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214568 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 02:09:28.218172 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214571 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 02:09:28.218172 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214574 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 02:09:28.218172 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214577 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 02:09:28.218172 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214586 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 02:09:28.218172 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214590 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 02:09:28.218172 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214596 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 02:09:28.218172 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214599 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 02:09:28.218172 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214602 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 02:09:28.218172 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214604 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 02:09:28.218172 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214608 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 02:09:28.218172 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214611 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 02:09:28.218172 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214613 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 02:09:28.218172 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214616 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 02:09:28.218172 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214619 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 02:09:28.218172 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214622 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 02:09:28.218172 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214624 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 02:09:28.218172 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214627 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 02:09:28.218172 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214632 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 02:09:28.218172 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214635 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 02:09:28.218172 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214638 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 02:09:28.218823 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214641 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 02:09:28.218823 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214644 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 02:09:28.218823 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214647 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 02:09:28.218823 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214649 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 02:09:28.218823 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214652 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 02:09:28.218823 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214655 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 02:09:28.218823 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214658 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 02:09:28.218823 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214660 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 02:09:28.218823 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214663 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 02:09:28.218823 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214666 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 02:09:28.218823 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214672 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 02:09:28.218823 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214675 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 02:09:28.218823 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214678 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 02:09:28.218823 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214681 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 02:09:28.218823 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214684 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 02:09:28.218823 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214687 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 02:09:28.218823 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214696 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 02:09:28.218823 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214699 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 02:09:28.218823 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214702 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 02:09:28.218823 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214705 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 02:09:28.219351 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:28.214708 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 02:09:28.219351 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.214714 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 21 02:09:28.219351 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.214918 2574 server.go:962] "Client rotation is on, will bootstrap in background" Jul 21 02:09:28.219351 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.217549 2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 21 02:09:28.219750 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.219735 2574 server.go:1019] "Starting client certificate rotation" Jul 21 02:09:28.219890 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.219860 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 21 02:09:28.219949 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.219925 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 21 02:09:28.223816 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.223797 2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 21 02:09:28.225313 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.225291 2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 21 02:09:28.238448 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.238431 2574 log.go:25] "Validated CRI v1 runtime API" Jul 21 02:09:28.240638 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.240625 2574 log.go:25] "Validated CRI v1 image API" Jul 21 02:09:28.243290 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.243226 2574 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 21 02:09:28.245801 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.245780 2574 fs.go:135] Filesystem UUIDs: map[09423834-c96b-40de-b60e-39208276ff28:/dev/nvme0n1p4 79016d8b-a8ce-43f7-b4b0-2bc7583c004a:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Jul 21 02:09:28.245858 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.245801 2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 21 02:09:28.248866 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.248845 2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 21 02:09:28.252045 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.251932 2574 manager.go:216] Machine: {Timestamp:2026-07-21 02:09:28.251415609 +0000 UTC m=+2.270909508 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097702 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ca8d64a9b93d1a62905b8b99a46dc SystemUUID:ec2ca8d6-4a9b-93d1-a629-05b8b99a46dc BootID:0fcbb346-8024-49c4-b1d8-aec48802de41 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:1b:e0:07:3f:bf Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:1b:e0:07:3f:bf Speed:0 Mtu:9001} {Name:ovs-system MacAddress:0a:3f:fc:63:22:7e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 21 02:09:28.252045 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.252038 2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 21 02:09:28.252176 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.252164 2574 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 21 02:09:28.255382 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.255333 2574 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 21 02:09:28.255533 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.255385 2574 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-37.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 21 02:09:28.256204 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.256192 2574 topology_manager.go:138] "Creating topology manager with none policy" Jul 21 02:09:28.256240 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.256206 2574 container_manager_linux.go:309] "Creating device plugin manager" Jul 21 02:09:28.256240 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.256215 2574 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 21 02:09:28.256240 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.256230 2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 21 02:09:28.257527 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.257516 2574 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 21 02:09:28.260419 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.260405 2574 state_mem.go:36] "Initialized new in-memory state store" Jul 21 02:09:28.260569 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.260558 2574 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 21 02:09:28.260605 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.260592 2574 kubelet.go:486] "Attempting to sync node with API server" Jul 21 02:09:28.260605 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.260603 2574 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 21 02:09:28.260669 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.260617 2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 21 02:09:28.260669 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.260629 2574 kubelet.go:398] "Adding apiserver pod source" Jul 21 02:09:28.260669 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.260644 2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 21 02:09:28.261076 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.261057 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 21 02:09:28.261143 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.261090 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 21 02:09:28.265826 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.265795 2574 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 21 02:09:28.267689 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.267673 2574 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 21 02:09:28.267751 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.267706 2574 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 21 02:09:28.268856 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.268843 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 21 02:09:28.268898 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.268863 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 21 02:09:28.268898 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.268873 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 21 02:09:28.268898 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.268883 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 21 02:09:28.268898 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.268897 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 21 02:09:28.269014 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.268906 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 21 02:09:28.269014 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.268918 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 21 02:09:28.269014 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.268926 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 21 02:09:28.269014 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.268936 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 21 02:09:28.269014 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.268942 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 21 02:09:28.269014 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.268952 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 21 02:09:28.269014 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.268957 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 21 02:09:28.269919 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.269907 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 21 02:09:28.269919 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.269918 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 21 02:09:28.271023 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:28.270994 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 21 02:09:28.271116 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:28.271062 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-37.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 21 02:09:28.273498 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.273484 2574 server.go:1268] "Started kubelet" Jul 21 02:09:28.273573 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.273542 2574 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 21 02:09:28.273698 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.273653 2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 21 02:09:28.273751 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.273738 2574 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 21 02:09:28.274847 ip-10-0-141-37 systemd[1]: Started Kubernetes Kubelet. Jul 21 02:09:28.275600 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.275445 2574 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 21 02:09:28.276205 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.276184 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gbn59" Jul 21 02:09:28.276818 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.276802 2574 server.go:310] "Adding debug handlers to kubelet server" Jul 21 02:09:28.279861 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:28.279840 2574 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 21 02:09:28.281450 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.281428 2574 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 21 02:09:28.281552 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.281441 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 21 02:09:28.281552 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:28.281522 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-37.ec2.internal\" not found" Jul 21 02:09:28.282260 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.282237 2574 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 21 02:09:28.282260 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.282259 2574 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 21 02:09:28.282458 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.282443 2574 reconstruct.go:97] "Volume reconstruction finished" Jul 21 02:09:28.282458 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.282456 2574 reconciler.go:29] "Reconciler: start to sync state" Jul 21 02:09:28.282578 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.282177 2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 21 02:09:28.282726 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.282707 2574 factory.go:153] Registering CRI-O factory Jul 21 02:09:28.282804 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.282729 2574 factory.go:223] Registration of the crio container factory successfully Jul 21 02:09:28.282804 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.282778 2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 21 02:09:28.282804 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.282788 2574 factory.go:55] Registering systemd factory Jul 21 02:09:28.282804 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.282794 2574 factory.go:223] Registration of the systemd container factory successfully Jul 21 02:09:28.282980 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.282815 2574 factory.go:103] Registering Raw factory Jul 21 02:09:28.282980 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.282825 2574 manager.go:1196] Started watching for new ooms in manager Jul 21 02:09:28.283213 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.283201 2574 manager.go:349] Starting recovery of all containers Jul 21 02:09:28.286197 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.286168 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gbn59" Jul 21 02:09:28.290829 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:28.290753 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 21 02:09:28.295911 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.295879 2574 manager.go:354] Recovery completed Jul 21 02:09:28.298274 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:28.298250 2574 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Jul 21 02:09:28.301681 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.301662 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 02:09:28.303712 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.303695 2574 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-141-37.ec2.internal" not found Jul 21 02:09:28.303803 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:28.303745 2574 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-141-37.ec2.internal\" not found" node="ip-10-0-141-37.ec2.internal" Jul 21 02:09:28.304012 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.303990 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-37.ec2.internal" event="NodeHasSufficientMemory" Jul 21 02:09:28.304125 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.304027 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-37.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 02:09:28.304125 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.304044 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-37.ec2.internal" event="NodeHasSufficientPID" Jul 21 02:09:28.304620 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.304601 2574 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 21 02:09:28.304620 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.304614 2574 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 21 02:09:28.304710 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.304631 2574 state_mem.go:36] "Initialized new in-memory state store" Jul 21 02:09:28.304710 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.304689 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 02:09:28.305492 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.305474 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-37.ec2.internal" event="NodeHasSufficientMemory" Jul 21 02:09:28.305578 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.305515 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-37.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 02:09:28.305578 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.305533 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-37.ec2.internal" event="NodeHasSufficientPID" Jul 21 02:09:28.306027 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.306013 2574 policy_none.go:49] "None policy: Start" Jul 21 02:09:28.306073 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.306032 2574 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 21 02:09:28.306073 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.306043 2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 21 02:09:28.307210 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.307199 2574 policy_none.go:47] "Start" Jul 21 02:09:28.321410 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.321151 2574 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-141-37.ec2.internal" not found Jul 21 02:09:28.348170 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.348147 2574 manager.go:337] "Starting Device Plugin manager" Jul 21 02:09:28.348303 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:28.348192 2574 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 21 02:09:28.348303 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.348205 2574 server.go:87] "Starting device plugin registration server" Jul 21 02:09:28.348546 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.348531 2574 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 21 02:09:28.348593 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.348550 2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 21 02:09:28.348733 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.348686 2574 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 21 02:09:28.348867 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.348776 2574 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 21 02:09:28.348867 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.348786 2574 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 21 02:09:28.349382 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:28.349361 2574 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 21 02:09:28.349471 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:28.349418 2574 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-37.ec2.internal\" not found" Jul 21 02:09:28.381517 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.381487 2574 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-141-37.ec2.internal" not found Jul 21 02:09:28.453471 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.453373 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 02:09:28.454493 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.454476 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-37.ec2.internal" event="NodeHasSufficientMemory" Jul 21 02:09:28.454586 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.454509 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-37.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 02:09:28.454586 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.454520 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-37.ec2.internal" event="NodeHasSufficientPID" Jul 21 02:09:28.454586 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.454549 2574 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-37.ec2.internal" Jul 21 02:09:28.465063 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.465046 2574 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-37.ec2.internal" Jul 21 02:09:28.465132 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:28.465068 2574 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-37.ec2.internal\": node \"ip-10-0-141-37.ec2.internal\" not found" Jul 21 02:09:28.476946 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.476920 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 21 02:09:28.478147 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.478132 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 21 02:09:28.478237 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.478187 2574 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 21 02:09:28.478237 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.478211 2574 kubelet.go:2448] "Starting kubelet main sync loop" Jul 21 02:09:28.478312 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:28.478244 2574 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 21 02:09:28.478999 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:28.478978 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-37.ec2.internal\" not found" Jul 21 02:09:28.480372 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.480353 2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 02:09:28.579208 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.579142 2574 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-37.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-37.ec2.internal"] Jul 21 02:09:28.579412 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.579251 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 02:09:28.579412 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:28.579164 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-37.ec2.internal\" not found" Jul 21 02:09:28.580417 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.580400 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-37.ec2.internal" event="NodeHasSufficientMemory" Jul 21 02:09:28.580524 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.580430 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-37.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 02:09:28.580524 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.580450 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-37.ec2.internal" event="NodeHasSufficientPID" Jul 21 02:09:28.583804 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.583784 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/96d07079af35c0ac51921ff8805ecb24-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-37.ec2.internal\" (UID: \"96d07079af35c0ac51921ff8805ecb24\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-37.ec2.internal" Jul 21 02:09:28.583854 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.583814 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/96d07079af35c0ac51921ff8805ecb24-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-37.ec2.internal\" (UID: \"96d07079af35c0ac51921ff8805ecb24\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-37.ec2.internal" Jul 21 02:09:28.584856 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.584843 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 02:09:28.584981 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.584965 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-37.ec2.internal" Jul 21 02:09:28.585023 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.585001 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 02:09:28.585662 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.585640 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-37.ec2.internal" event="NodeHasSufficientMemory" Jul 21 02:09:28.585763 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.585669 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-37.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 02:09:28.585763 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.585679 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-37.ec2.internal" event="NodeHasSufficientPID" Jul 21 02:09:28.585763 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.585643 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-37.ec2.internal" event="NodeHasSufficientMemory" Jul 21 02:09:28.585763 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.585758 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-37.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 02:09:28.585908 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.585771 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-37.ec2.internal" event="NodeHasSufficientPID" Jul 21 02:09:28.587617 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.587600 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-37.ec2.internal" Jul 21 02:09:28.587707 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.587629 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 02:09:28.588342 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.588329 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-37.ec2.internal" event="NodeHasSufficientMemory" Jul 21 02:09:28.588408 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.588359 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-37.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 02:09:28.588408 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.588373 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-37.ec2.internal" event="NodeHasSufficientPID" Jul 21 02:09:28.611282 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:28.611245 2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-37.ec2.internal\" not found" node="ip-10-0-141-37.ec2.internal" Jul 21 02:09:28.616466 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:28.616446 2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-37.ec2.internal\" not found" node="ip-10-0-141-37.ec2.internal" Jul 21 02:09:28.679473 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:28.679424 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-37.ec2.internal\" not found" Jul 21 02:09:28.684013 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.683990 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/96d07079af35c0ac51921ff8805ecb24-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-37.ec2.internal\" (UID: \"96d07079af35c0ac51921ff8805ecb24\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-37.ec2.internal" Jul 21 02:09:28.684111 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.684026 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/03a5161441a8d59f0580132e9ff32a06-config\") pod \"kube-apiserver-proxy-ip-10-0-141-37.ec2.internal\" (UID: \"03a5161441a8d59f0580132e9ff32a06\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-37.ec2.internal" Jul 21 02:09:28.684111 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.684050 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/96d07079af35c0ac51921ff8805ecb24-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-37.ec2.internal\" (UID: \"96d07079af35c0ac51921ff8805ecb24\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-37.ec2.internal" Jul 21 02:09:28.684111 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.684087 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/96d07079af35c0ac51921ff8805ecb24-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-37.ec2.internal\" (UID: \"96d07079af35c0ac51921ff8805ecb24\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-37.ec2.internal" Jul 21 02:09:28.684215 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.684087 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/96d07079af35c0ac51921ff8805ecb24-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-37.ec2.internal\" (UID: \"96d07079af35c0ac51921ff8805ecb24\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-37.ec2.internal" Jul 21 02:09:28.779968 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:28.779853 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-37.ec2.internal\" not found" Jul 21 02:09:28.785246 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.785222 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/03a5161441a8d59f0580132e9ff32a06-config\") pod \"kube-apiserver-proxy-ip-10-0-141-37.ec2.internal\" (UID: \"03a5161441a8d59f0580132e9ff32a06\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-37.ec2.internal" Jul 21 02:09:28.785373 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.785266 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/03a5161441a8d59f0580132e9ff32a06-config\") pod \"kube-apiserver-proxy-ip-10-0-141-37.ec2.internal\" (UID: \"03a5161441a8d59f0580132e9ff32a06\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-37.ec2.internal" Jul 21 02:09:28.880065 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:28.880030 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-37.ec2.internal\" not found" Jul 21 02:09:28.916329 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.916294 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-37.ec2.internal" Jul 21 02:09:28.918766 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:28.918742 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-37.ec2.internal" Jul 21 02:09:28.980868 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:28.980821 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-37.ec2.internal\" not found" Jul 21 02:09:29.088476 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:29.088397 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-37.ec2.internal\" not found" Jul 21 02:09:29.188587 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:29.188549 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-37.ec2.internal\" not found" Jul 21 02:09:29.204794 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.204763 2574 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 02:09:29.219775 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.219743 2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 21 02:09:29.219928 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.219908 2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 21 02:09:29.219990 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.219934 2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 21 02:09:29.261678 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.261651 2574 apiserver.go:52] "Watching apiserver" Jul 21 02:09:29.277034 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.277008 2574 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 02:09:29.279285 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.279258 2574 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 21 02:09:29.280859 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.280677 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-x2w8b","openshift-multus/network-metrics-daemon-8v67q","openshift-network-diagnostics/network-check-target-qqnx4","openshift-network-operator/iptables-alerter-75hhp","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsl9k","openshift-cluster-node-tuning-operator/tuned-lgkbl","openshift-dns/node-resolver-8d69j","openshift-image-registry/node-ca-b9rfh","openshift-multus/multus-lkmsc","openshift-ovn-kubernetes/ovnkube-node-qv9kr","kube-system/global-pull-secret-syncer-ljdcc","kube-system/konnectivity-agent-wj24l"] Jul 21 02:09:29.281875 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.281855 2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-37.ec2.internal" Jul 21 02:09:29.281983 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.281875 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 21 02:09:29.282881 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.282859 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-x2w8b" Jul 21 02:09:29.284069 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.284039 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8v67q" Jul 21 02:09:29.284168 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:29.284104 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8v67q" podUID="03e524b0-40bd-46b4-ac8f-eb346e94a7dc" Jul 21 02:09:29.285427 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.285408 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqnx4" Jul 21 02:09:29.285515 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:29.285462 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qqnx4" podUID="80deee90-4b47-4cdf-bc87-f9242ad63cec" Jul 21 02:09:29.285977 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.285908 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 21 02:09:29.285977 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.285909 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 21 02:09:29.286163 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.286129 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 21 02:09:29.286229 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.286185 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 21 02:09:29.286229 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.286201 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5ctmr\"" Jul 21 02:09:29.286540 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.286522 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 21 02:09:29.287075 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.287052 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-75hhp" Jul 21 02:09:29.288549 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.288530 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsl9k" Jul 21 02:09:29.288695 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.288673 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3504ec09-d4b6-41ce-b77d-d6c73b437a33-cnibin\") pod \"multus-additional-cni-plugins-x2w8b\" (UID: \"3504ec09-d4b6-41ce-b77d-d6c73b437a33\") " pod="openshift-multus/multus-additional-cni-plugins-x2w8b" Jul 21 02:09:29.288766 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.288709 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3504ec09-d4b6-41ce-b77d-d6c73b437a33-os-release\") pod \"multus-additional-cni-plugins-x2w8b\" (UID: \"3504ec09-d4b6-41ce-b77d-d6c73b437a33\") " pod="openshift-multus/multus-additional-cni-plugins-x2w8b" Jul 21 02:09:29.288766 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.288742 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3504ec09-d4b6-41ce-b77d-d6c73b437a33-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x2w8b\" (UID: \"3504ec09-d4b6-41ce-b77d-d6c73b437a33\") " pod="openshift-multus/multus-additional-cni-plugins-x2w8b" Jul 21 02:09:29.288873 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.288797 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3504ec09-d4b6-41ce-b77d-d6c73b437a33-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x2w8b\" (UID: \"3504ec09-d4b6-41ce-b77d-d6c73b437a33\") " pod="openshift-multus/multus-additional-cni-plugins-x2w8b" Jul 21 02:09:29.288873 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.288845 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6ljh\" (UniqueName: \"kubernetes.io/projected/3504ec09-d4b6-41ce-b77d-d6c73b437a33-kube-api-access-d6ljh\") pod \"multus-additional-cni-plugins-x2w8b\" (UID: \"3504ec09-d4b6-41ce-b77d-d6c73b437a33\") " pod="openshift-multus/multus-additional-cni-plugins-x2w8b" Jul 21 02:09:29.288957 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.288902 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3504ec09-d4b6-41ce-b77d-d6c73b437a33-system-cni-dir\") pod \"multus-additional-cni-plugins-x2w8b\" (UID: \"3504ec09-d4b6-41ce-b77d-d6c73b437a33\") " pod="openshift-multus/multus-additional-cni-plugins-x2w8b" Jul 21 02:09:29.288957 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.288939 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3504ec09-d4b6-41ce-b77d-d6c73b437a33-cni-binary-copy\") pod \"multus-additional-cni-plugins-x2w8b\" (UID: \"3504ec09-d4b6-41ce-b77d-d6c73b437a33\") " pod="openshift-multus/multus-additional-cni-plugins-x2w8b" Jul 21 02:09:29.289047 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.288970 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3504ec09-d4b6-41ce-b77d-d6c73b437a33-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x2w8b\" (UID: \"3504ec09-d4b6-41ce-b77d-d6c73b437a33\") " pod="openshift-multus/multus-additional-cni-plugins-x2w8b" Jul 21 02:09:29.289047 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.289003 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03e524b0-40bd-46b4-ac8f-eb346e94a7dc-metrics-certs\") pod \"network-metrics-daemon-8v67q\" (UID: \"03e524b0-40bd-46b4-ac8f-eb346e94a7dc\") " pod="openshift-multus/network-metrics-daemon-8v67q" Jul 21 02:09:29.289047 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.289028 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtqtv\" (UniqueName: \"kubernetes.io/projected/03e524b0-40bd-46b4-ac8f-eb346e94a7dc-kube-api-access-mtqtv\") pod \"network-metrics-daemon-8v67q\" (UID: \"03e524b0-40bd-46b4-ac8f-eb346e94a7dc\") " pod="openshift-multus/network-metrics-daemon-8v67q" Jul 21 02:09:29.289384 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.289367 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 21 02:09:29.289611 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.289587 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 21 02:09:29.289752 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.289729 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 21 02:09:29.289859 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.289845 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2m2rw\"" Jul 21 02:09:29.289934 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.289908 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-20 02:04:28 +0000 UTC" deadline="2028-05-03 23:18:50.493081984 +0000 UTC" Jul 21 02:09:29.289980 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.289937 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15669h9m21.203148968s" Jul 21 02:09:29.290014 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.290005 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lgkbl" Jul 21 02:09:29.290713 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.290698 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 21 02:09:29.291005 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.290992 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 21 02:09:29.291056 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.291040 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 21 02:09:29.291056 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.291048 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-kb5bp\"" Jul 21 02:09:29.291260 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.291242 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8d69j" Jul 21 02:09:29.292427 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.292390 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 21 02:09:29.292631 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.292616 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-b9rfh" Jul 21 02:09:29.292736 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.292710 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4mvn8\"" Jul 21 02:09:29.292882 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.292856 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 21 02:09:29.293449 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.293434 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-m4xgl\"" Jul 21 02:09:29.293791 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.293777 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 21 02:09:29.294319 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.294303 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lkmsc" Jul 21 02:09:29.294419 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.294401 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 21 02:09:29.295928 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.295912 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" Jul 21 02:09:29.297304 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.297282 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 21 02:09:29.297413 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.297306 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 21 02:09:29.297413 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.297313 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 21 02:09:29.297413 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.297315 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 21 02:09:29.297580 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.297418 2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 21 02:09:29.297580 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.297473 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-vvhbv\"" Jul 21 02:09:29.297580 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.297515 2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-37.ec2.internal" Jul 21 02:09:29.297744 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.297729 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ljdcc" Jul 21 02:09:29.297796 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:29.297781 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ljdcc" podUID="4bee3776-d7db-4be5-8e76-337ee2b89187" Jul 21 02:09:29.298984 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.298964 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-37.ec2.internal"] Jul 21 02:09:29.299069 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.299053 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wj24l" Jul 21 02:09:29.301948 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.301727 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 21 02:09:29.302196 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.302158 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 21 02:09:29.302288 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.302263 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-vwf59\"" Jul 21 02:09:29.302393 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.302369 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 21 02:09:29.303076 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.302492 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 21 02:09:29.303076 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.302613 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4mfn6\"" Jul 21 02:09:29.303076 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.302773 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 21 02:09:29.303076 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.302917 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kz6wr\"" Jul 21 02:09:29.303076 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.302959 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 21 02:09:29.303406 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.303289 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 21 02:09:29.303458 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.303417 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 21 02:09:29.305435 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.305415 2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 21 02:09:29.309713 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.309693 2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 21 02:09:29.309805 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.309695 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-141-37.ec2.internal"] Jul 21 02:09:29.340084 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.340021 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-kb6mm" Jul 21 02:09:29.349610 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.349580 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-kb6mm" Jul 21 02:09:29.383506 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.383474 2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 21 02:09:29.390241 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.390217 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3504ec09-d4b6-41ce-b77d-d6c73b437a33-os-release\") pod \"multus-additional-cni-plugins-x2w8b\" (UID: \"3504ec09-d4b6-41ce-b77d-d6c73b437a33\") " pod="openshift-multus/multus-additional-cni-plugins-x2w8b" Jul 21 02:09:29.390461 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.390259 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r474z\" (UniqueName: \"kubernetes.io/projected/c8e442c1-e98a-4af5-9274-5ab79f671aef-kube-api-access-r474z\") pod \"node-resolver-8d69j\" (UID: \"c8e442c1-e98a-4af5-9274-5ab79f671aef\") " pod="openshift-dns/node-resolver-8d69j" Jul 21 02:09:29.390461 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.390306 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/83fe72d8-5aa8-4232-9317-3e0d06bc9535-host\") pod \"node-ca-b9rfh\" (UID: \"83fe72d8-5aa8-4232-9317-3e0d06bc9535\") " pod="openshift-image-registry/node-ca-b9rfh" Jul 21 02:09:29.390461 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.390354 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9-run-openvswitch\") pod \"ovnkube-node-qv9kr\" (UID: \"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" Jul 21 02:09:29.390461 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.390385 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3504ec09-d4b6-41ce-b77d-d6c73b437a33-os-release\") pod \"multus-additional-cni-plugins-x2w8b\" (UID: \"3504ec09-d4b6-41ce-b77d-d6c73b437a33\") " pod="openshift-multus/multus-additional-cni-plugins-x2w8b" Jul 21 02:09:29.390461 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.390389 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9-ovn-node-metrics-cert\") pod \"ovnkube-node-qv9kr\" (UID: \"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" Jul 21 02:09:29.390461 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.390426 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/48bafec4-325f-45e3-b658-5e5a9ff57629-device-dir\") pod \"aws-ebs-csi-driver-node-qsl9k\" (UID: \"48bafec4-325f-45e3-b658-5e5a9ff57629\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsl9k" Jul 21 02:09:29.390461 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.390446 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/91f30d91-79e3-44be-a679-ad1b6e873a99-etc-sysctl-conf\") pod \"tuned-lgkbl\" (UID: \"91f30d91-79e3-44be-a679-ad1b6e873a99\") " pod="openshift-cluster-node-tuning-operator/tuned-lgkbl" Jul 21 02:09:29.390461 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.390462 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91f30d91-79e3-44be-a679-ad1b6e873a99-tmp\") pod \"tuned-lgkbl\" (UID: \"91f30d91-79e3-44be-a679-ad1b6e873a99\") " pod="openshift-cluster-node-tuning-operator/tuned-lgkbl" Jul 21 02:09:29.390770 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.390479 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9-run-systemd\") pod \"ovnkube-node-qv9kr\" (UID: \"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" Jul 21 02:09:29.390770 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.390495 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7zpj\" (UniqueName: \"kubernetes.io/projected/bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9-kube-api-access-z7zpj\") pod \"ovnkube-node-qv9kr\" (UID: \"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" Jul 21 02:09:29.390770 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.390541 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/48bafec4-325f-45e3-b658-5e5a9ff57629-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qsl9k\" (UID: \"48bafec4-325f-45e3-b658-5e5a9ff57629\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsl9k" Jul 21 02:09:29.390770 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.390562 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/48bafec4-325f-45e3-b658-5e5a9ff57629-socket-dir\") pod \"aws-ebs-csi-driver-node-qsl9k\" (UID: \"48bafec4-325f-45e3-b658-5e5a9ff57629\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsl9k" Jul 21 02:09:29.390770 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.390591 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/df138c44-9b99-4291-9996-d10aee5562ff-iptables-alerter-script\") pod \"iptables-alerter-75hhp\" (UID: \"df138c44-9b99-4291-9996-d10aee5562ff\") " pod="openshift-network-operator/iptables-alerter-75hhp" Jul 21 02:09:29.390770 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.390619 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51277326-7122-47d0-991c-4fe0c5aa0102-system-cni-dir\") pod \"multus-lkmsc\" (UID: \"51277326-7122-47d0-991c-4fe0c5aa0102\") " pod="openshift-multus/multus-lkmsc" Jul 21 02:09:29.390770 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.390635 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/51277326-7122-47d0-991c-4fe0c5aa0102-multus-conf-dir\") pod \"multus-lkmsc\" (UID: \"51277326-7122-47d0-991c-4fe0c5aa0102\") " pod="openshift-multus/multus-lkmsc" Jul 21 02:09:29.390770 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.390655 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9-etc-openvswitch\") pod \"ovnkube-node-qv9kr\" (UID: \"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" Jul 21 02:09:29.390770 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.390682 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9-node-log\") pod \"ovnkube-node-qv9kr\" (UID: \"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" Jul 21 02:09:29.390770 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.390707 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/91f30d91-79e3-44be-a679-ad1b6e873a99-etc-tuned\") pod \"tuned-lgkbl\" (UID: \"91f30d91-79e3-44be-a679-ad1b6e873a99\") " pod="openshift-cluster-node-tuning-operator/tuned-lgkbl" Jul 21 02:09:29.390770 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.390731 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw5wm\" (UniqueName: \"kubernetes.io/projected/83fe72d8-5aa8-4232-9317-3e0d06bc9535-kube-api-access-sw5wm\") pod \"node-ca-b9rfh\" (UID: \"83fe72d8-5aa8-4232-9317-3e0d06bc9535\") " pod="openshift-image-registry/node-ca-b9rfh" Jul 21 02:09:29.391188 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.390779 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51277326-7122-47d0-991c-4fe0c5aa0102-host-var-lib-kubelet\") pod \"multus-lkmsc\" (UID: \"51277326-7122-47d0-991c-4fe0c5aa0102\") " pod="openshift-multus/multus-lkmsc" Jul 21 02:09:29.391188 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.390832 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9-env-overrides\") pod \"ovnkube-node-qv9kr\" (UID: \"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" Jul 21 02:09:29.391188 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.390890 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e0411e85-4230-45d9-9055-1e2edf4cd322-konnectivity-ca\") pod \"konnectivity-agent-wj24l\" (UID: \"e0411e85-4230-45d9-9055-1e2edf4cd322\") " pod="kube-system/konnectivity-agent-wj24l" Jul 21 02:09:29.391188 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.390952 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/48bafec4-325f-45e3-b658-5e5a9ff57629-registration-dir\") pod \"aws-ebs-csi-driver-node-qsl9k\" (UID: \"48bafec4-325f-45e3-b658-5e5a9ff57629\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsl9k" Jul 21 02:09:29.391188 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.390979 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/48bafec4-325f-45e3-b658-5e5a9ff57629-etc-selinux\") pod \"aws-ebs-csi-driver-node-qsl9k\" (UID: \"48bafec4-325f-45e3-b658-5e5a9ff57629\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsl9k" Jul 21 02:09:29.391188 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.391005 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/51277326-7122-47d0-991c-4fe0c5aa0102-host-run-k8s-cni-cncf-io\") pod \"multus-lkmsc\" (UID: \"51277326-7122-47d0-991c-4fe0c5aa0102\") " pod="openshift-multus/multus-lkmsc" Jul 21 02:09:29.391188 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.391026 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/51277326-7122-47d0-991c-4fe0c5aa0102-multus-daemon-config\") pod \"multus-lkmsc\" (UID: \"51277326-7122-47d0-991c-4fe0c5aa0102\") " pod="openshift-multus/multus-lkmsc" Jul 21 02:09:29.391188 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.391049 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/51277326-7122-47d0-991c-4fe0c5aa0102-os-release\") pod \"multus-lkmsc\" (UID: \"51277326-7122-47d0-991c-4fe0c5aa0102\") " pod="openshift-multus/multus-lkmsc" Jul 21 02:09:29.391188 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.391076 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/51277326-7122-47d0-991c-4fe0c5aa0102-multus-socket-dir-parent\") pod \"multus-lkmsc\" (UID: \"51277326-7122-47d0-991c-4fe0c5aa0102\") " pod="openshift-multus/multus-lkmsc" Jul 21 02:09:29.391188 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.391124 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht26w\" (UniqueName: \"kubernetes.io/projected/80deee90-4b47-4cdf-bc87-f9242ad63cec-kube-api-access-ht26w\") pod \"network-check-target-qqnx4\" (UID: \"80deee90-4b47-4cdf-bc87-f9242ad63cec\") " pod="openshift-network-diagnostics/network-check-target-qqnx4" Jul 21 02:09:29.391188 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.391162 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/48bafec4-325f-45e3-b658-5e5a9ff57629-sys-fs\") pod \"aws-ebs-csi-driver-node-qsl9k\" (UID: \"48bafec4-325f-45e3-b658-5e5a9ff57629\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsl9k" Jul 21 02:09:29.391772 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.391230 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/91f30d91-79e3-44be-a679-ad1b6e873a99-sys\") pod \"tuned-lgkbl\" (UID: \"91f30d91-79e3-44be-a679-ad1b6e873a99\") " pod="openshift-cluster-node-tuning-operator/tuned-lgkbl" Jul 21 02:09:29.391772 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.391275 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91f30d91-79e3-44be-a679-ad1b6e873a99-var-lib-kubelet\") pod \"tuned-lgkbl\" (UID: \"91f30d91-79e3-44be-a679-ad1b6e873a99\") " pod="openshift-cluster-node-tuning-operator/tuned-lgkbl" Jul 21 02:09:29.391772 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.391331 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51277326-7122-47d0-991c-4fe0c5aa0102-etc-kubernetes\") pod \"multus-lkmsc\" (UID: \"51277326-7122-47d0-991c-4fe0c5aa0102\") " pod="openshift-multus/multus-lkmsc" Jul 21 02:09:29.391772 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.391360 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9-host-cni-bin\") pod \"ovnkube-node-qv9kr\" (UID: \"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" Jul 21 02:09:29.391772 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.391392 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3504ec09-d4b6-41ce-b77d-d6c73b437a33-cni-binary-copy\") pod \"multus-additional-cni-plugins-x2w8b\" (UID: \"3504ec09-d4b6-41ce-b77d-d6c73b437a33\") " pod="openshift-multus/multus-additional-cni-plugins-x2w8b" Jul 21 02:09:29.391772 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.391420 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51277326-7122-47d0-991c-4fe0c5aa0102-multus-cni-dir\") pod \"multus-lkmsc\" (UID: \"51277326-7122-47d0-991c-4fe0c5aa0102\") " pod="openshift-multus/multus-lkmsc" Jul 21 02:09:29.391772 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.391445 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/51277326-7122-47d0-991c-4fe0c5aa0102-cni-binary-copy\") pod \"multus-lkmsc\" (UID: \"51277326-7122-47d0-991c-4fe0c5aa0102\") " pod="openshift-multus/multus-lkmsc" Jul 21 02:09:29.391772 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.391472 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/51277326-7122-47d0-991c-4fe0c5aa0102-host-var-lib-cni-bin\") pod \"multus-lkmsc\" (UID: \"51277326-7122-47d0-991c-4fe0c5aa0102\") " pod="openshift-multus/multus-lkmsc" Jul 21 02:09:29.391772 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.391502 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/51277326-7122-47d0-991c-4fe0c5aa0102-host-run-multus-certs\") pod \"multus-lkmsc\" (UID: \"51277326-7122-47d0-991c-4fe0c5aa0102\") " pod="openshift-multus/multus-lkmsc" Jul 21 02:09:29.391772 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.391550 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3504ec09-d4b6-41ce-b77d-d6c73b437a33-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x2w8b\" (UID: \"3504ec09-d4b6-41ce-b77d-d6c73b437a33\") " pod="openshift-multus/multus-additional-cni-plugins-x2w8b" Jul 21 02:09:29.391772 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.391581 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03e524b0-40bd-46b4-ac8f-eb346e94a7dc-metrics-certs\") pod \"network-metrics-daemon-8v67q\" (UID: \"03e524b0-40bd-46b4-ac8f-eb346e94a7dc\") " pod="openshift-multus/network-metrics-daemon-8v67q" Jul 21 02:09:29.391772 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.391608 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9-host-kubelet\") pod \"ovnkube-node-qv9kr\" (UID: \"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" Jul 21 02:09:29.391772 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.391645 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4bee3776-d7db-4be5-8e76-337ee2b89187-original-pull-secret\") pod \"global-pull-secret-syncer-ljdcc\" (UID: \"4bee3776-d7db-4be5-8e76-337ee2b89187\") " pod="kube-system/global-pull-secret-syncer-ljdcc" Jul 21 02:09:29.391772 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.391687 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/91f30d91-79e3-44be-a679-ad1b6e873a99-etc-sysctl-d\") pod \"tuned-lgkbl\" (UID: \"91f30d91-79e3-44be-a679-ad1b6e873a99\") " pod="openshift-cluster-node-tuning-operator/tuned-lgkbl" Jul 21 02:09:29.391772 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.391716 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c8e442c1-e98a-4af5-9274-5ab79f671aef-hosts-file\") pod \"node-resolver-8d69j\" (UID: \"c8e442c1-e98a-4af5-9274-5ab79f671aef\") " pod="openshift-dns/node-resolver-8d69j" Jul 21 02:09:29.391772 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.391743 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/df138c44-9b99-4291-9996-d10aee5562ff-host-slash\") pod \"iptables-alerter-75hhp\" (UID: \"df138c44-9b99-4291-9996-d10aee5562ff\") " pod="openshift-network-operator/iptables-alerter-75hhp" Jul 21 02:09:29.391772 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.391770 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/51277326-7122-47d0-991c-4fe0c5aa0102-host-run-netns\") pod \"multus-lkmsc\" (UID: \"51277326-7122-47d0-991c-4fe0c5aa0102\") " pod="openshift-multus/multus-lkmsc" Jul 21 02:09:29.392510 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:29.391778 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 02:09:29.392510 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.391794 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9-host-slash\") pod \"ovnkube-node-qv9kr\" (UID: \"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" Jul 21 02:09:29.392510 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.391819 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9-var-lib-openvswitch\") pod \"ovnkube-node-qv9kr\" (UID: \"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" Jul 21 02:09:29.392510 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.391871 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qv9kr\" (UID: \"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" Jul 21 02:09:29.392510 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:29.391931 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03e524b0-40bd-46b4-ac8f-eb346e94a7dc-metrics-certs podName:03e524b0-40bd-46b4-ac8f-eb346e94a7dc nodeName:}" failed. No retries permitted until 2026-07-21 02:09:29.891885091 +0000 UTC m=+3.911378980 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03e524b0-40bd-46b4-ac8f-eb346e94a7dc-metrics-certs") pod "network-metrics-daemon-8v67q" (UID: "03e524b0-40bd-46b4-ac8f-eb346e94a7dc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 02:09:29.392510 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.391959 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4bee3776-d7db-4be5-8e76-337ee2b89187-kubelet-config\") pod \"global-pull-secret-syncer-ljdcc\" (UID: \"4bee3776-d7db-4be5-8e76-337ee2b89187\") " pod="kube-system/global-pull-secret-syncer-ljdcc" Jul 21 02:09:29.392510 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.392008 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/91f30d91-79e3-44be-a679-ad1b6e873a99-host\") pod \"tuned-lgkbl\" (UID: \"91f30d91-79e3-44be-a679-ad1b6e873a99\") " pod="openshift-cluster-node-tuning-operator/tuned-lgkbl" Jul 21 02:09:29.392510 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.392048 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9q8l\" (UniqueName: \"kubernetes.io/projected/df138c44-9b99-4291-9996-d10aee5562ff-kube-api-access-v9q8l\") pod \"iptables-alerter-75hhp\" (UID: \"df138c44-9b99-4291-9996-d10aee5562ff\") " pod="openshift-network-operator/iptables-alerter-75hhp" Jul 21 02:09:29.392510 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.392073 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/51277326-7122-47d0-991c-4fe0c5aa0102-cnibin\") pod \"multus-lkmsc\" (UID: \"51277326-7122-47d0-991c-4fe0c5aa0102\") " pod="openshift-multus/multus-lkmsc" Jul 21 02:09:29.392510 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.392096 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/51277326-7122-47d0-991c-4fe0c5aa0102-hostroot\") pod \"multus-lkmsc\" (UID: \"51277326-7122-47d0-991c-4fe0c5aa0102\") " pod="openshift-multus/multus-lkmsc" Jul 21 02:09:29.392510 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.392123 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3504ec09-d4b6-41ce-b77d-d6c73b437a33-cni-binary-copy\") pod \"multus-additional-cni-plugins-x2w8b\" (UID: \"3504ec09-d4b6-41ce-b77d-d6c73b437a33\") " pod="openshift-multus/multus-additional-cni-plugins-x2w8b" Jul 21 02:09:29.392510 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.392122 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6mrj\" (UniqueName: \"kubernetes.io/projected/51277326-7122-47d0-991c-4fe0c5aa0102-kube-api-access-f6mrj\") pod \"multus-lkmsc\" (UID: \"51277326-7122-47d0-991c-4fe0c5aa0102\") " pod="openshift-multus/multus-lkmsc" Jul 21 02:09:29.392510 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.392180 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9-host-cni-netd\") pod \"ovnkube-node-qv9kr\" (UID: \"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" Jul 21 02:09:29.392510 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.392217 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3504ec09-d4b6-41ce-b77d-d6c73b437a33-system-cni-dir\") pod \"multus-additional-cni-plugins-x2w8b\" (UID: \"3504ec09-d4b6-41ce-b77d-d6c73b437a33\") " pod="openshift-multus/multus-additional-cni-plugins-x2w8b" Jul 21 02:09:29.392510 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.392210 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3504ec09-d4b6-41ce-b77d-d6c73b437a33-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x2w8b\" (UID: \"3504ec09-d4b6-41ce-b77d-d6c73b437a33\") " pod="openshift-multus/multus-additional-cni-plugins-x2w8b" Jul 21 02:09:29.392510 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.392249 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtqtv\" (UniqueName: \"kubernetes.io/projected/03e524b0-40bd-46b4-ac8f-eb346e94a7dc-kube-api-access-mtqtv\") pod \"network-metrics-daemon-8v67q\" (UID: \"03e524b0-40bd-46b4-ac8f-eb346e94a7dc\") " pod="openshift-multus/network-metrics-daemon-8v67q" Jul 21 02:09:29.393110 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.392277 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/51277326-7122-47d0-991c-4fe0c5aa0102-host-var-lib-cni-multus\") pod \"multus-lkmsc\" (UID: \"51277326-7122-47d0-991c-4fe0c5aa0102\") " pod="openshift-multus/multus-lkmsc" Jul 21 02:09:29.393110 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.392332 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3504ec09-d4b6-41ce-b77d-d6c73b437a33-system-cni-dir\") pod \"multus-additional-cni-plugins-x2w8b\" (UID: \"3504ec09-d4b6-41ce-b77d-d6c73b437a33\") " pod="openshift-multus/multus-additional-cni-plugins-x2w8b" Jul 21 02:09:29.393110 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.392363 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9-systemd-units\") pod \"ovnkube-node-qv9kr\" (UID: \"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" Jul 21 02:09:29.393110 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.392386 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4bee3776-d7db-4be5-8e76-337ee2b89187-dbus\") pod \"global-pull-secret-syncer-ljdcc\" (UID: \"4bee3776-d7db-4be5-8e76-337ee2b89187\") " pod="kube-system/global-pull-secret-syncer-ljdcc" Jul 21 02:09:29.393110 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.392411 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e0411e85-4230-45d9-9055-1e2edf4cd322-agent-certs\") pod \"konnectivity-agent-wj24l\" (UID: \"e0411e85-4230-45d9-9055-1e2edf4cd322\") " pod="kube-system/konnectivity-agent-wj24l" Jul 21 02:09:29.393110 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.392437 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3504ec09-d4b6-41ce-b77d-d6c73b437a33-cnibin\") pod \"multus-additional-cni-plugins-x2w8b\" (UID: \"3504ec09-d4b6-41ce-b77d-d6c73b437a33\") " pod="openshift-multus/multus-additional-cni-plugins-x2w8b" Jul 21 02:09:29.393110 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.392474 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3504ec09-d4b6-41ce-b77d-d6c73b437a33-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x2w8b\" (UID: \"3504ec09-d4b6-41ce-b77d-d6c73b437a33\") " pod="openshift-multus/multus-additional-cni-plugins-x2w8b" Jul 21 02:09:29.393110 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.392550 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3504ec09-d4b6-41ce-b77d-d6c73b437a33-cnibin\") pod \"multus-additional-cni-plugins-x2w8b\" (UID: \"3504ec09-d4b6-41ce-b77d-d6c73b437a33\") " pod="openshift-multus/multus-additional-cni-plugins-x2w8b" Jul 21 02:09:29.393110 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.392566 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6ljh\" (UniqueName: \"kubernetes.io/projected/3504ec09-d4b6-41ce-b77d-d6c73b437a33-kube-api-access-d6ljh\") pod \"multus-additional-cni-plugins-x2w8b\" (UID: \"3504ec09-d4b6-41ce-b77d-d6c73b437a33\") " pod="openshift-multus/multus-additional-cni-plugins-x2w8b" Jul 21 02:09:29.393110 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.392601 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/91f30d91-79e3-44be-a679-ad1b6e873a99-etc-modprobe-d\") pod \"tuned-lgkbl\" (UID: \"91f30d91-79e3-44be-a679-ad1b6e873a99\") " pod="openshift-cluster-node-tuning-operator/tuned-lgkbl" Jul 21 02:09:29.393110 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.392627 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/91f30d91-79e3-44be-a679-ad1b6e873a99-etc-systemd\") pod \"tuned-lgkbl\" (UID: \"91f30d91-79e3-44be-a679-ad1b6e873a99\") " pod="openshift-cluster-node-tuning-operator/tuned-lgkbl" Jul 21 02:09:29.393110 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.392629 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3504ec09-d4b6-41ce-b77d-d6c73b437a33-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x2w8b\" (UID: \"3504ec09-d4b6-41ce-b77d-d6c73b437a33\") " pod="openshift-multus/multus-additional-cni-plugins-x2w8b" Jul 21 02:09:29.393110 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.392655 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9-host-run-netns\") pod \"ovnkube-node-qv9kr\" (UID: \"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" Jul 21 02:09:29.393110 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.392676 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9-run-ovn\") pod \"ovnkube-node-qv9kr\" (UID: \"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" Jul 21 02:09:29.393110 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.392712 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9-ovnkube-config\") pod \"ovnkube-node-qv9kr\" (UID: \"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" Jul 21 02:09:29.393110 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.392755 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3504ec09-d4b6-41ce-b77d-d6c73b437a33-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x2w8b\" (UID: \"3504ec09-d4b6-41ce-b77d-d6c73b437a33\") " pod="openshift-multus/multus-additional-cni-plugins-x2w8b" Jul 21 02:09:29.393661 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.392775 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91f30d91-79e3-44be-a679-ad1b6e873a99-etc-kubernetes\") pod \"tuned-lgkbl\" (UID: \"91f30d91-79e3-44be-a679-ad1b6e873a99\") " pod="openshift-cluster-node-tuning-operator/tuned-lgkbl" Jul 21 02:09:29.393661 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.392799 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/91f30d91-79e3-44be-a679-ad1b6e873a99-lib-modules\") pod \"tuned-lgkbl\" (UID: \"91f30d91-79e3-44be-a679-ad1b6e873a99\") " pod="openshift-cluster-node-tuning-operator/tuned-lgkbl" Jul 21 02:09:29.393661 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.392823 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8rtl\" (UniqueName: \"kubernetes.io/projected/48bafec4-325f-45e3-b658-5e5a9ff57629-kube-api-access-q8rtl\") pod \"aws-ebs-csi-driver-node-qsl9k\" (UID: \"48bafec4-325f-45e3-b658-5e5a9ff57629\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsl9k" Jul 21 02:09:29.393661 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.392857 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/91f30d91-79e3-44be-a679-ad1b6e873a99-run\") pod \"tuned-lgkbl\" (UID: \"91f30d91-79e3-44be-a679-ad1b6e873a99\") " pod="openshift-cluster-node-tuning-operator/tuned-lgkbl" Jul 21 02:09:29.393661 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.392897 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl5wh\" (UniqueName: \"kubernetes.io/projected/91f30d91-79e3-44be-a679-ad1b6e873a99-kube-api-access-cl5wh\") pod \"tuned-lgkbl\" (UID: \"91f30d91-79e3-44be-a679-ad1b6e873a99\") " pod="openshift-cluster-node-tuning-operator/tuned-lgkbl" Jul 21 02:09:29.393661 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.392922 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8e442c1-e98a-4af5-9274-5ab79f671aef-tmp-dir\") pod \"node-resolver-8d69j\" (UID: \"c8e442c1-e98a-4af5-9274-5ab79f671aef\") " pod="openshift-dns/node-resolver-8d69j" Jul 21 02:09:29.393661 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.392944 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/83fe72d8-5aa8-4232-9317-3e0d06bc9535-serviceca\") pod \"node-ca-b9rfh\" (UID: \"83fe72d8-5aa8-4232-9317-3e0d06bc9535\") " pod="openshift-image-registry/node-ca-b9rfh" Jul 21 02:09:29.393661 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.392966 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/91f30d91-79e3-44be-a679-ad1b6e873a99-etc-sysconfig\") pod \"tuned-lgkbl\" (UID: \"91f30d91-79e3-44be-a679-ad1b6e873a99\") " pod="openshift-cluster-node-tuning-operator/tuned-lgkbl" Jul 21 02:09:29.393661 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.392987 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9-log-socket\") pod \"ovnkube-node-qv9kr\" (UID: \"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" Jul 21 02:09:29.393661 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.393009 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9-host-run-ovn-kubernetes\") pod \"ovnkube-node-qv9kr\" (UID: \"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" Jul 21 02:09:29.393661 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.393031 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9-ovnkube-script-lib\") pod \"ovnkube-node-qv9kr\" (UID: \"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" Jul 21 02:09:29.393661 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.393186 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3504ec09-d4b6-41ce-b77d-d6c73b437a33-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x2w8b\" (UID: \"3504ec09-d4b6-41ce-b77d-d6c73b437a33\") " pod="openshift-multus/multus-additional-cni-plugins-x2w8b" Jul 21 02:09:29.401976 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.401946 2574 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 21 02:09:29.405280 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.405258 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtqtv\" (UniqueName: \"kubernetes.io/projected/03e524b0-40bd-46b4-ac8f-eb346e94a7dc-kube-api-access-mtqtv\") pod \"network-metrics-daemon-8v67q\" (UID: \"03e524b0-40bd-46b4-ac8f-eb346e94a7dc\") " pod="openshift-multus/network-metrics-daemon-8v67q" Jul 21 02:09:29.405925 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.405904 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6ljh\" (UniqueName: \"kubernetes.io/projected/3504ec09-d4b6-41ce-b77d-d6c73b437a33-kube-api-access-d6ljh\") pod \"multus-additional-cni-plugins-x2w8b\" (UID: \"3504ec09-d4b6-41ce-b77d-d6c73b437a33\") " pod="openshift-multus/multus-additional-cni-plugins-x2w8b" Jul 21 02:09:29.493428 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.493392 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/91f30d91-79e3-44be-a679-ad1b6e873a99-etc-modprobe-d\") pod \"tuned-lgkbl\" (UID: \"91f30d91-79e3-44be-a679-ad1b6e873a99\") " pod="openshift-cluster-node-tuning-operator/tuned-lgkbl" Jul 21 02:09:29.493428 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.493430 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/91f30d91-79e3-44be-a679-ad1b6e873a99-etc-systemd\") pod \"tuned-lgkbl\" (UID: \"91f30d91-79e3-44be-a679-ad1b6e873a99\") " pod="openshift-cluster-node-tuning-operator/tuned-lgkbl" Jul 21 02:09:29.493428 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.493449 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9-host-run-netns\") pod \"ovnkube-node-qv9kr\" (UID: \"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" Jul 21 02:09:29.493731 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.493487 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9-run-ovn\") pod \"ovnkube-node-qv9kr\" (UID: \"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" Jul 21 02:09:29.493731 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.493512 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9-ovnkube-config\") pod \"ovnkube-node-qv9kr\" (UID: \"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" Jul 21 02:09:29.493731 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.493529 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91f30d91-79e3-44be-a679-ad1b6e873a99-etc-kubernetes\") pod \"tuned-lgkbl\" (UID: \"91f30d91-79e3-44be-a679-ad1b6e873a99\") " pod="openshift-cluster-node-tuning-operator/tuned-lgkbl" Jul 21 02:09:29.493731 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.493551 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/91f30d91-79e3-44be-a679-ad1b6e873a99-lib-modules\") pod \"tuned-lgkbl\" (UID: \"91f30d91-79e3-44be-a679-ad1b6e873a99\") " pod="openshift-cluster-node-tuning-operator/tuned-lgkbl" Jul 21 02:09:29.493731 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.493544 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/91f30d91-79e3-44be-a679-ad1b6e873a99-etc-systemd\") pod \"tuned-lgkbl\" (UID: \"91f30d91-79e3-44be-a679-ad1b6e873a99\") " pod="openshift-cluster-node-tuning-operator/tuned-lgkbl" Jul 21 02:09:29.493731 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.493619 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91f30d91-79e3-44be-a679-ad1b6e873a99-etc-kubernetes\") pod \"tuned-lgkbl\" (UID: \"91f30d91-79e3-44be-a679-ad1b6e873a99\") " pod="openshift-cluster-node-tuning-operator/tuned-lgkbl" Jul 21 02:09:29.493731 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.493627 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9-run-ovn\") pod \"ovnkube-node-qv9kr\" (UID: \"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" Jul 21 02:09:29.493731 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.493654 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8rtl\" (UniqueName: \"kubernetes.io/projected/48bafec4-325f-45e3-b658-5e5a9ff57629-kube-api-access-q8rtl\") pod \"aws-ebs-csi-driver-node-qsl9k\" (UID: \"48bafec4-325f-45e3-b658-5e5a9ff57629\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsl9k" Jul 21 02:09:29.493731 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.493659 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/91f30d91-79e3-44be-a679-ad1b6e873a99-etc-modprobe-d\") pod \"tuned-lgkbl\" (UID: \"91f30d91-79e3-44be-a679-ad1b6e873a99\") " pod="openshift-cluster-node-tuning-operator/tuned-lgkbl" Jul 21 02:09:29.493731 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.493683 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/91f30d91-79e3-44be-a679-ad1b6e873a99-run\") pod \"tuned-lgkbl\" (UID: \"91f30d91-79e3-44be-a679-ad1b6e873a99\") " pod="openshift-cluster-node-tuning-operator/tuned-lgkbl" Jul 21 02:09:29.493731 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.493710 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl5wh\" (UniqueName: \"kubernetes.io/projected/91f30d91-79e3-44be-a679-ad1b6e873a99-kube-api-access-cl5wh\") pod \"tuned-lgkbl\" (UID: \"91f30d91-79e3-44be-a679-ad1b6e873a99\") " pod="openshift-cluster-node-tuning-operator/tuned-lgkbl" Jul 21 02:09:29.494298 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.493758 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8e442c1-e98a-4af5-9274-5ab79f671aef-tmp-dir\") pod \"node-resolver-8d69j\" (UID: \"c8e442c1-e98a-4af5-9274-5ab79f671aef\") " pod="openshift-dns/node-resolver-8d69j" Jul 21 02:09:29.494298 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.493715 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9-host-run-netns\") pod \"ovnkube-node-qv9kr\" (UID: \"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" Jul 21 02:09:29.494298 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.493787 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/83fe72d8-5aa8-4232-9317-3e0d06bc9535-serviceca\") pod \"node-ca-b9rfh\" (UID: \"83fe72d8-5aa8-4232-9317-3e0d06bc9535\") " pod="openshift-image-registry/node-ca-b9rfh" Jul 21 02:09:29.494298 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.493718 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/91f30d91-79e3-44be-a679-ad1b6e873a99-lib-modules\") pod \"tuned-lgkbl\" (UID: \"91f30d91-79e3-44be-a679-ad1b6e873a99\") " pod="openshift-cluster-node-tuning-operator/tuned-lgkbl" Jul 21 02:09:29.494298 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.493812 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/91f30d91-79e3-44be-a679-ad1b6e873a99-etc-sysconfig\") pod \"tuned-lgkbl\" (UID: \"91f30d91-79e3-44be-a679-ad1b6e873a99\") " pod="openshift-cluster-node-tuning-operator/tuned-lgkbl" Jul 21 02:09:29.494298 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.493799 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/91f30d91-79e3-44be-a679-ad1b6e873a99-run\") pod \"tuned-lgkbl\" (UID: \"91f30d91-79e3-44be-a679-ad1b6e873a99\") " pod="openshift-cluster-node-tuning-operator/tuned-lgkbl" Jul 21 02:09:29.494298 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.493836 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9-log-socket\") pod \"ovnkube-node-qv9kr\" (UID: \"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" Jul 21 02:09:29.494298 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.493861 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9-host-run-ovn-kubernetes\") pod \"ovnkube-node-qv9kr\" (UID: \"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" Jul 21 02:09:29.494298 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.493897 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9-ovnkube-script-lib\") pod \"ovnkube-node-qv9kr\" (UID: \"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" Jul 21 02:09:29.494298 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.493923 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r474z\" (UniqueName: \"kubernetes.io/projected/c8e442c1-e98a-4af5-9274-5ab79f671aef-kube-api-access-r474z\") pod \"node-resolver-8d69j\" (UID: \"c8e442c1-e98a-4af5-9274-5ab79f671aef\") " pod="openshift-dns/node-resolver-8d69j" Jul 21 02:09:29.494298 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.493945 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/83fe72d8-5aa8-4232-9317-3e0d06bc9535-host\") pod \"node-ca-b9rfh\" (UID: \"83fe72d8-5aa8-4232-9317-3e0d06bc9535\") " pod="openshift-image-registry/node-ca-b9rfh" Jul 21 02:09:29.494298 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.493969 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9-run-openvswitch\") pod \"ovnkube-node-qv9kr\" (UID: \"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" Jul 21 02:09:29.494298 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.493996 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9-ovn-node-metrics-cert\") pod \"ovnkube-node-qv9kr\" (UID: \"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" Jul 21 02:09:29.494298 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.494010 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9-host-run-ovn-kubernetes\") pod \"ovnkube-node-qv9kr\" (UID: \"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" Jul 21 02:09:29.494298 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.494022 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/48bafec4-325f-45e3-b658-5e5a9ff57629-device-dir\") pod \"aws-ebs-csi-driver-node-qsl9k\" (UID: \"48bafec4-325f-45e3-b658-5e5a9ff57629\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsl9k" Jul 21 02:09:29.494298 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.494047 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/91f30d91-79e3-44be-a679-ad1b6e873a99-etc-sysctl-conf\") pod \"tuned-lgkbl\" (UID: \"91f30d91-79e3-44be-a679-ad1b6e873a99\") " pod="openshift-cluster-node-tuning-operator/tuned-lgkbl" Jul 21 02:09:29.494298 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.494071 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9-log-socket\") pod \"ovnkube-node-qv9kr\" (UID: \"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" Jul 21 02:09:29.494298 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.494083 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91f30d91-79e3-44be-a679-ad1b6e873a99-tmp\") pod \"tuned-lgkbl\" (UID: \"91f30d91-79e3-44be-a679-ad1b6e873a99\") " pod="openshift-cluster-node-tuning-operator/tuned-lgkbl" Jul 21 02:09:29.495333 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.494088 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9-run-openvswitch\") pod \"ovnkube-node-qv9kr\" (UID: \"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" Jul 21 02:09:29.495333 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.494108 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9-run-systemd\") pod \"ovnkube-node-qv9kr\" (UID: \"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" Jul 21 02:09:29.495333 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.494131 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8e442c1-e98a-4af5-9274-5ab79f671aef-tmp-dir\") pod \"node-resolver-8d69j\" (UID: \"c8e442c1-e98a-4af5-9274-5ab79f671aef\") " pod="openshift-dns/node-resolver-8d69j" Jul 21 02:09:29.495333 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.494136 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7zpj\" (UniqueName: \"kubernetes.io/projected/bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9-kube-api-access-z7zpj\") pod \"ovnkube-node-qv9kr\" (UID: \"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" Jul 21 02:09:29.495333 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.494148 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/83fe72d8-5aa8-4232-9317-3e0d06bc9535-host\") pod \"node-ca-b9rfh\" (UID: \"83fe72d8-5aa8-4232-9317-3e0d06bc9535\") " pod="openshift-image-registry/node-ca-b9rfh" Jul 21 02:09:29.495333 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.494171 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/48bafec4-325f-45e3-b658-5e5a9ff57629-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qsl9k\" (UID: \"48bafec4-325f-45e3-b658-5e5a9ff57629\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsl9k" Jul 21 02:09:29.495333 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.494197 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/48bafec4-325f-45e3-b658-5e5a9ff57629-socket-dir\") pod \"aws-ebs-csi-driver-node-qsl9k\" (UID: \"48bafec4-325f-45e3-b658-5e5a9ff57629\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsl9k" Jul 21 02:09:29.495333 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.494222 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/df138c44-9b99-4291-9996-d10aee5562ff-iptables-alerter-script\") pod \"iptables-alerter-75hhp\" (UID: \"df138c44-9b99-4291-9996-d10aee5562ff\") " pod="openshift-network-operator/iptables-alerter-75hhp" Jul 21 02:09:29.495333 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.494249 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51277326-7122-47d0-991c-4fe0c5aa0102-system-cni-dir\") pod \"multus-lkmsc\" (UID: \"51277326-7122-47d0-991c-4fe0c5aa0102\") " pod="openshift-multus/multus-lkmsc" Jul 21 02:09:29.495333 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.494276 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/51277326-7122-47d0-991c-4fe0c5aa0102-multus-conf-dir\") pod \"multus-lkmsc\" (UID: \"51277326-7122-47d0-991c-4fe0c5aa0102\") " pod="openshift-multus/multus-lkmsc" Jul 21 02:09:29.495333 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.494306 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9-etc-openvswitch\") pod \"ovnkube-node-qv9kr\" (UID: \"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" Jul 21 02:09:29.495333 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.494330 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9-node-log\") pod \"ovnkube-node-qv9kr\" (UID: \"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" Jul 21 02:09:29.495333 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.494373 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/91f30d91-79e3-44be-a679-ad1b6e873a99-etc-tuned\") pod \"tuned-lgkbl\" (UID: \"91f30d91-79e3-44be-a679-ad1b6e873a99\") " pod="openshift-cluster-node-tuning-operator/tuned-lgkbl" Jul 21 02:09:29.495333 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.494401 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/91f30d91-79e3-44be-a679-ad1b6e873a99-etc-sysconfig\") pod \"tuned-lgkbl\" (UID: \"91f30d91-79e3-44be-a679-ad1b6e873a99\") " pod="openshift-cluster-node-tuning-operator/tuned-lgkbl" Jul 21 02:09:29.495333 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.494404 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw5wm\" (UniqueName: \"kubernetes.io/projected/83fe72d8-5aa8-4232-9317-3e0d06bc9535-kube-api-access-sw5wm\") pod \"node-ca-b9rfh\" (UID: \"83fe72d8-5aa8-4232-9317-3e0d06bc9535\") " pod="openshift-image-registry/node-ca-b9rfh" Jul 21 02:09:29.495333 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.494434 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9-ovnkube-config\") pod \"ovnkube-node-qv9kr\" (UID: \"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" Jul 21 02:09:29.495333 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.494458 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51277326-7122-47d0-991c-4fe0c5aa0102-host-var-lib-kubelet\") pod \"multus-lkmsc\" (UID: \"51277326-7122-47d0-991c-4fe0c5aa0102\") " pod="openshift-multus/multus-lkmsc" Jul 21 02:09:29.495333 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.494465 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/48bafec4-325f-45e3-b658-5e5a9ff57629-socket-dir\") pod \"aws-ebs-csi-driver-node-qsl9k\" (UID: \"48bafec4-325f-45e3-b658-5e5a9ff57629\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsl9k" Jul 21 02:09:29.496268 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.494500 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9-env-overrides\") pod \"ovnkube-node-qv9kr\" (UID: \"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" Jul 21 02:09:29.496268 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.494506 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51277326-7122-47d0-991c-4fe0c5aa0102-host-var-lib-kubelet\") pod \"multus-lkmsc\" (UID: \"51277326-7122-47d0-991c-4fe0c5aa0102\") " pod="openshift-multus/multus-lkmsc" Jul 21 02:09:29.496268 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.494315 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/48bafec4-325f-45e3-b658-5e5a9ff57629-device-dir\") pod \"aws-ebs-csi-driver-node-qsl9k\" (UID: \"48bafec4-325f-45e3-b658-5e5a9ff57629\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsl9k" Jul 21 02:09:29.496268 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.494521 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9-etc-openvswitch\") pod \"ovnkube-node-qv9kr\" (UID: \"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" Jul 21 02:09:29.496268 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.494529 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e0411e85-4230-45d9-9055-1e2edf4cd322-konnectivity-ca\") pod \"konnectivity-agent-wj24l\" (UID: \"e0411e85-4230-45d9-9055-1e2edf4cd322\") " pod="kube-system/konnectivity-agent-wj24l" Jul 21 02:09:29.496268 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.494560 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/48bafec4-325f-45e3-b658-5e5a9ff57629-registration-dir\") pod \"aws-ebs-csi-driver-node-qsl9k\" (UID: \"48bafec4-325f-45e3-b658-5e5a9ff57629\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsl9k" Jul 21 02:09:29.496268 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.494567 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9-node-log\") pod \"ovnkube-node-qv9kr\" (UID: \"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" Jul 21 02:09:29.496268 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.494584 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/48bafec4-325f-45e3-b658-5e5a9ff57629-etc-selinux\") pod \"aws-ebs-csi-driver-node-qsl9k\" (UID: \"48bafec4-325f-45e3-b658-5e5a9ff57629\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsl9k" Jul 21 02:09:29.496268 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.494342 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/83fe72d8-5aa8-4232-9317-3e0d06bc9535-serviceca\") pod \"node-ca-b9rfh\" (UID: \"83fe72d8-5aa8-4232-9317-3e0d06bc9535\") " pod="openshift-image-registry/node-ca-b9rfh" Jul 21 02:09:29.496268 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.494665 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/51277326-7122-47d0-991c-4fe0c5aa0102-host-run-k8s-cni-cncf-io\") pod \"multus-lkmsc\" (UID: \"51277326-7122-47d0-991c-4fe0c5aa0102\") " pod="openshift-multus/multus-lkmsc" Jul 21 02:09:29.496268 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.494674 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51277326-7122-47d0-991c-4fe0c5aa0102-system-cni-dir\") pod \"multus-lkmsc\" (UID: \"51277326-7122-47d0-991c-4fe0c5aa0102\") " pod="openshift-multus/multus-lkmsc" Jul 21 02:09:29.496268 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.494661 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/91f30d91-79e3-44be-a679-ad1b6e873a99-etc-sysctl-conf\") pod \"tuned-lgkbl\" (UID: \"91f30d91-79e3-44be-a679-ad1b6e873a99\") " pod="openshift-cluster-node-tuning-operator/tuned-lgkbl" Jul 21 02:09:29.496268 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.494708 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/51277326-7122-47d0-991c-4fe0c5aa0102-multus-conf-dir\") pod \"multus-lkmsc\" (UID: \"51277326-7122-47d0-991c-4fe0c5aa0102\") " pod="openshift-multus/multus-lkmsc" Jul 21 02:09:29.496268 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.494783 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9-ovnkube-script-lib\") pod \"ovnkube-node-qv9kr\" (UID: \"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" Jul 21 02:09:29.496268 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.494851 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/51277326-7122-47d0-991c-4fe0c5aa0102-multus-daemon-config\") pod \"multus-lkmsc\" (UID: \"51277326-7122-47d0-991c-4fe0c5aa0102\") " pod="openshift-multus/multus-lkmsc" Jul 21 02:09:29.496268 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.494886 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/51277326-7122-47d0-991c-4fe0c5aa0102-os-release\") pod \"multus-lkmsc\" (UID: \"51277326-7122-47d0-991c-4fe0c5aa0102\") " pod="openshift-multus/multus-lkmsc" Jul 21 02:09:29.496268 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.494925 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/51277326-7122-47d0-991c-4fe0c5aa0102-multus-socket-dir-parent\") pod \"multus-lkmsc\" (UID: \"51277326-7122-47d0-991c-4fe0c5aa0102\") " pod="openshift-multus/multus-lkmsc" Jul 21 02:09:29.496268 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.494955 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht26w\" (UniqueName: \"kubernetes.io/projected/80deee90-4b47-4cdf-bc87-f9242ad63cec-kube-api-access-ht26w\") pod \"network-check-target-qqnx4\" (UID: \"80deee90-4b47-4cdf-bc87-f9242ad63cec\") " pod="openshift-network-diagnostics/network-check-target-qqnx4" Jul 21 02:09:29.497048 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.494959 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/48bafec4-325f-45e3-b658-5e5a9ff57629-registration-dir\") pod \"aws-ebs-csi-driver-node-qsl9k\" (UID: \"48bafec4-325f-45e3-b658-5e5a9ff57629\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsl9k" Jul 21 02:09:29.497048 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.494990 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/48bafec4-325f-45e3-b658-5e5a9ff57629-sys-fs\") pod \"aws-ebs-csi-driver-node-qsl9k\" (UID: \"48bafec4-325f-45e3-b658-5e5a9ff57629\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsl9k" Jul 21 02:09:29.497048 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.495021 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/48bafec4-325f-45e3-b658-5e5a9ff57629-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qsl9k\" (UID: \"48bafec4-325f-45e3-b658-5e5a9ff57629\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsl9k" Jul 21 02:09:29.497048 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.495026 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9-env-overrides\") pod \"ovnkube-node-qv9kr\" (UID: \"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" Jul 21 02:09:29.497048 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.495073 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/91f30d91-79e3-44be-a679-ad1b6e873a99-sys\") pod \"tuned-lgkbl\" (UID: \"91f30d91-79e3-44be-a679-ad1b6e873a99\") " pod="openshift-cluster-node-tuning-operator/tuned-lgkbl" Jul 21 02:09:29.497048 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.495103 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91f30d91-79e3-44be-a679-ad1b6e873a99-var-lib-kubelet\") pod \"tuned-lgkbl\" (UID: \"91f30d91-79e3-44be-a679-ad1b6e873a99\") " pod="openshift-cluster-node-tuning-operator/tuned-lgkbl" Jul 21 02:09:29.497048 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.495118 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/48bafec4-325f-45e3-b658-5e5a9ff57629-sys-fs\") pod \"aws-ebs-csi-driver-node-qsl9k\" (UID: \"48bafec4-325f-45e3-b658-5e5a9ff57629\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsl9k" Jul 21 02:09:29.497048 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.495132 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51277326-7122-47d0-991c-4fe0c5aa0102-etc-kubernetes\") pod \"multus-lkmsc\" (UID: \"51277326-7122-47d0-991c-4fe0c5aa0102\") " pod="openshift-multus/multus-lkmsc" Jul 21 02:09:29.497048 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.495171 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/df138c44-9b99-4291-9996-d10aee5562ff-iptables-alerter-script\") pod \"iptables-alerter-75hhp\" (UID: \"df138c44-9b99-4291-9996-d10aee5562ff\") " pod="openshift-network-operator/iptables-alerter-75hhp" Jul 21 02:09:29.497048 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.495182 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9-host-cni-bin\") pod \"ovnkube-node-qv9kr\" (UID: \"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" Jul 21 02:09:29.497048 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.495294 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51277326-7122-47d0-991c-4fe0c5aa0102-multus-cni-dir\") pod \"multus-lkmsc\" (UID: \"51277326-7122-47d0-991c-4fe0c5aa0102\") " pod="openshift-multus/multus-lkmsc" Jul 21 02:09:29.497048 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.495327 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/51277326-7122-47d0-991c-4fe0c5aa0102-cni-binary-copy\") pod \"multus-lkmsc\" (UID: \"51277326-7122-47d0-991c-4fe0c5aa0102\") " pod="openshift-multus/multus-lkmsc" Jul 21 02:09:29.497048 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.495373 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/51277326-7122-47d0-991c-4fe0c5aa0102-host-var-lib-cni-bin\") pod \"multus-lkmsc\" (UID: \"51277326-7122-47d0-991c-4fe0c5aa0102\") " pod="openshift-multus/multus-lkmsc" Jul 21 02:09:29.497048 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.495387 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51277326-7122-47d0-991c-4fe0c5aa0102-etc-kubernetes\") pod \"multus-lkmsc\" (UID: \"51277326-7122-47d0-991c-4fe0c5aa0102\") " pod="openshift-multus/multus-lkmsc" Jul 21 02:09:29.497048 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.495399 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/51277326-7122-47d0-991c-4fe0c5aa0102-host-run-multus-certs\") pod \"multus-lkmsc\" (UID: \"51277326-7122-47d0-991c-4fe0c5aa0102\") " pod="openshift-multus/multus-lkmsc" Jul 21 02:09:29.497048 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.495458 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9-host-cni-bin\") pod \"ovnkube-node-qv9kr\" (UID: \"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" Jul 21 02:09:29.497048 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.495462 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/51277326-7122-47d0-991c-4fe0c5aa0102-host-run-multus-certs\") pod \"multus-lkmsc\" (UID: \"51277326-7122-47d0-991c-4fe0c5aa0102\") " pod="openshift-multus/multus-lkmsc" Jul 21 02:09:29.497048 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.495500 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9-run-systemd\") pod \"ovnkube-node-qv9kr\" (UID: \"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" Jul 21 02:09:29.497760 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.495512 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/51277326-7122-47d0-991c-4fe0c5aa0102-multus-socket-dir-parent\") pod \"multus-lkmsc\" (UID: \"51277326-7122-47d0-991c-4fe0c5aa0102\") " pod="openshift-multus/multus-lkmsc" Jul 21 02:09:29.497760 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.495537 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/51277326-7122-47d0-991c-4fe0c5aa0102-host-run-k8s-cni-cncf-io\") pod \"multus-lkmsc\" (UID: \"51277326-7122-47d0-991c-4fe0c5aa0102\") " pod="openshift-multus/multus-lkmsc" Jul 21 02:09:29.497760 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.495570 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/51277326-7122-47d0-991c-4fe0c5aa0102-os-release\") pod \"multus-lkmsc\" (UID: \"51277326-7122-47d0-991c-4fe0c5aa0102\") " pod="openshift-multus/multus-lkmsc" Jul 21 02:09:29.497760 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.495581 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/51277326-7122-47d0-991c-4fe0c5aa0102-multus-daemon-config\") pod \"multus-lkmsc\" (UID: \"51277326-7122-47d0-991c-4fe0c5aa0102\") " pod="openshift-multus/multus-lkmsc" Jul 21 02:09:29.497760 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.495617 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91f30d91-79e3-44be-a679-ad1b6e873a99-var-lib-kubelet\") pod \"tuned-lgkbl\" (UID: \"91f30d91-79e3-44be-a679-ad1b6e873a99\") " pod="openshift-cluster-node-tuning-operator/tuned-lgkbl" Jul 21 02:09:29.497760 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.495663 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/91f30d91-79e3-44be-a679-ad1b6e873a99-sys\") pod \"tuned-lgkbl\" (UID: \"91f30d91-79e3-44be-a679-ad1b6e873a99\") " pod="openshift-cluster-node-tuning-operator/tuned-lgkbl" Jul 21 02:09:29.497760 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.495666 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51277326-7122-47d0-991c-4fe0c5aa0102-multus-cni-dir\") pod \"multus-lkmsc\" (UID: \"51277326-7122-47d0-991c-4fe0c5aa0102\") " pod="openshift-multus/multus-lkmsc" Jul 21 02:09:29.497760 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.495684 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/48bafec4-325f-45e3-b658-5e5a9ff57629-etc-selinux\") pod \"aws-ebs-csi-driver-node-qsl9k\" (UID: \"48bafec4-325f-45e3-b658-5e5a9ff57629\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsl9k" Jul 21 02:09:29.497760 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.495708 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e0411e85-4230-45d9-9055-1e2edf4cd322-konnectivity-ca\") pod \"konnectivity-agent-wj24l\" (UID: \"e0411e85-4230-45d9-9055-1e2edf4cd322\") " pod="kube-system/konnectivity-agent-wj24l" Jul 21 02:09:29.497760 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.495734 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9-host-kubelet\") pod \"ovnkube-node-qv9kr\" (UID: \"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" Jul 21 02:09:29.497760 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.495752 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/51277326-7122-47d0-991c-4fe0c5aa0102-host-var-lib-cni-bin\") pod \"multus-lkmsc\" (UID: \"51277326-7122-47d0-991c-4fe0c5aa0102\") " pod="openshift-multus/multus-lkmsc" Jul 21 02:09:29.497760 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.495769 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4bee3776-d7db-4be5-8e76-337ee2b89187-original-pull-secret\") pod \"global-pull-secret-syncer-ljdcc\" (UID: \"4bee3776-d7db-4be5-8e76-337ee2b89187\") " pod="kube-system/global-pull-secret-syncer-ljdcc" Jul 21 02:09:29.497760 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.495800 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/91f30d91-79e3-44be-a679-ad1b6e873a99-etc-sysctl-d\") pod \"tuned-lgkbl\" (UID: \"91f30d91-79e3-44be-a679-ad1b6e873a99\") " pod="openshift-cluster-node-tuning-operator/tuned-lgkbl" Jul 21 02:09:29.497760 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.495836 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c8e442c1-e98a-4af5-9274-5ab79f671aef-hosts-file\") pod \"node-resolver-8d69j\" (UID: \"c8e442c1-e98a-4af5-9274-5ab79f671aef\") " pod="openshift-dns/node-resolver-8d69j" Jul 21 02:09:29.497760 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.495887 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c8e442c1-e98a-4af5-9274-5ab79f671aef-hosts-file\") pod \"node-resolver-8d69j\" (UID: \"c8e442c1-e98a-4af5-9274-5ab79f671aef\") " pod="openshift-dns/node-resolver-8d69j" Jul 21 02:09:29.497760 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.495886 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/df138c44-9b99-4291-9996-d10aee5562ff-host-slash\") pod \"iptables-alerter-75hhp\" (UID: \"df138c44-9b99-4291-9996-d10aee5562ff\") " pod="openshift-network-operator/iptables-alerter-75hhp" Jul 21 02:09:29.497760 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.495897 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/91f30d91-79e3-44be-a679-ad1b6e873a99-etc-sysctl-d\") pod \"tuned-lgkbl\" (UID: \"91f30d91-79e3-44be-a679-ad1b6e873a99\") " pod="openshift-cluster-node-tuning-operator/tuned-lgkbl" Jul 21 02:09:29.497760 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.495930 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/51277326-7122-47d0-991c-4fe0c5aa0102-host-run-netns\") pod \"multus-lkmsc\" (UID: \"51277326-7122-47d0-991c-4fe0c5aa0102\") " pod="openshift-multus/multus-lkmsc" Jul 21 02:09:29.498389 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.495931 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9-host-kubelet\") pod \"ovnkube-node-qv9kr\" (UID: \"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" Jul 21 02:09:29.498389 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.495937 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/df138c44-9b99-4291-9996-d10aee5562ff-host-slash\") pod \"iptables-alerter-75hhp\" (UID: \"df138c44-9b99-4291-9996-d10aee5562ff\") " pod="openshift-network-operator/iptables-alerter-75hhp" Jul 21 02:09:29.498389 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.495972 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9-host-slash\") pod \"ovnkube-node-qv9kr\" (UID: \"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" Jul 21 02:09:29.498389 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.496002 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9-var-lib-openvswitch\") pod \"ovnkube-node-qv9kr\" (UID: \"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" Jul 21 02:09:29.498389 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.496008 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9-host-slash\") pod \"ovnkube-node-qv9kr\" (UID: \"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" Jul 21 02:09:29.498389 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:29.496014 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 02:09:29.498389 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.496034 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qv9kr\" (UID: \"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" Jul 21 02:09:29.498389 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.496046 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/51277326-7122-47d0-991c-4fe0c5aa0102-host-run-netns\") pod \"multus-lkmsc\" (UID: \"51277326-7122-47d0-991c-4fe0c5aa0102\") " pod="openshift-multus/multus-lkmsc" Jul 21 02:09:29.498389 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.496069 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4bee3776-d7db-4be5-8e76-337ee2b89187-kubelet-config\") pod \"global-pull-secret-syncer-ljdcc\" (UID: \"4bee3776-d7db-4be5-8e76-337ee2b89187\") " pod="kube-system/global-pull-secret-syncer-ljdcc" Jul 21 02:09:29.498389 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.496102 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/91f30d91-79e3-44be-a679-ad1b6e873a99-host\") pod \"tuned-lgkbl\" (UID: \"91f30d91-79e3-44be-a679-ad1b6e873a99\") " pod="openshift-cluster-node-tuning-operator/tuned-lgkbl" Jul 21 02:09:29.498389 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.496118 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9-var-lib-openvswitch\") pod \"ovnkube-node-qv9kr\" (UID: \"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" Jul 21 02:09:29.498389 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:29.496123 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bee3776-d7db-4be5-8e76-337ee2b89187-original-pull-secret podName:4bee3776-d7db-4be5-8e76-337ee2b89187 nodeName:}" failed. No retries permitted until 2026-07-21 02:09:29.996105332 +0000 UTC m=+4.015599238 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4bee3776-d7db-4be5-8e76-337ee2b89187-original-pull-secret") pod "global-pull-secret-syncer-ljdcc" (UID: "4bee3776-d7db-4be5-8e76-337ee2b89187") : object "kube-system"/"original-pull-secret" not registered Jul 21 02:09:29.498389 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.496145 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/91f30d91-79e3-44be-a679-ad1b6e873a99-host\") pod \"tuned-lgkbl\" (UID: \"91f30d91-79e3-44be-a679-ad1b6e873a99\") " pod="openshift-cluster-node-tuning-operator/tuned-lgkbl" Jul 21 02:09:29.498389 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.496161 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9q8l\" (UniqueName: \"kubernetes.io/projected/df138c44-9b99-4291-9996-d10aee5562ff-kube-api-access-v9q8l\") pod \"iptables-alerter-75hhp\" (UID: \"df138c44-9b99-4291-9996-d10aee5562ff\") " pod="openshift-network-operator/iptables-alerter-75hhp" Jul 21 02:09:29.498389 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.496188 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/51277326-7122-47d0-991c-4fe0c5aa0102-cnibin\") pod \"multus-lkmsc\" (UID: \"51277326-7122-47d0-991c-4fe0c5aa0102\") " pod="openshift-multus/multus-lkmsc" Jul 21 02:09:29.498389 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.496213 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/51277326-7122-47d0-991c-4fe0c5aa0102-hostroot\") pod \"multus-lkmsc\" (UID: \"51277326-7122-47d0-991c-4fe0c5aa0102\") " pod="openshift-multus/multus-lkmsc" Jul 21 02:09:29.498389 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.496225 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4bee3776-d7db-4be5-8e76-337ee2b89187-kubelet-config\") pod \"global-pull-secret-syncer-ljdcc\" (UID: \"4bee3776-d7db-4be5-8e76-337ee2b89187\") " pod="kube-system/global-pull-secret-syncer-ljdcc" Jul 21 02:09:29.499335 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.496238 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6mrj\" (UniqueName: \"kubernetes.io/projected/51277326-7122-47d0-991c-4fe0c5aa0102-kube-api-access-f6mrj\") pod \"multus-lkmsc\" (UID: \"51277326-7122-47d0-991c-4fe0c5aa0102\") " pod="openshift-multus/multus-lkmsc" Jul 21 02:09:29.499335 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.496263 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9-host-cni-netd\") pod \"ovnkube-node-qv9kr\" (UID: \"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" Jul 21 02:09:29.499335 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.496273 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/51277326-7122-47d0-991c-4fe0c5aa0102-cnibin\") pod \"multus-lkmsc\" (UID: \"51277326-7122-47d0-991c-4fe0c5aa0102\") " pod="openshift-multus/multus-lkmsc" Jul 21 02:09:29.499335 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.496292 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/51277326-7122-47d0-991c-4fe0c5aa0102-host-var-lib-cni-multus\") pod \"multus-lkmsc\" (UID: \"51277326-7122-47d0-991c-4fe0c5aa0102\") " pod="openshift-multus/multus-lkmsc" Jul 21 02:09:29.499335 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.496295 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/51277326-7122-47d0-991c-4fe0c5aa0102-cni-binary-copy\") pod \"multus-lkmsc\" (UID: \"51277326-7122-47d0-991c-4fe0c5aa0102\") " pod="openshift-multus/multus-lkmsc" Jul 21 02:09:29.499335 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.496187 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qv9kr\" (UID: \"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" Jul 21 02:09:29.499335 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.496349 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/51277326-7122-47d0-991c-4fe0c5aa0102-host-var-lib-cni-multus\") pod \"multus-lkmsc\" (UID: \"51277326-7122-47d0-991c-4fe0c5aa0102\") " pod="openshift-multus/multus-lkmsc" Jul 21 02:09:29.499335 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.496394 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/51277326-7122-47d0-991c-4fe0c5aa0102-hostroot\") pod \"multus-lkmsc\" (UID: \"51277326-7122-47d0-991c-4fe0c5aa0102\") " pod="openshift-multus/multus-lkmsc" Jul 21 02:09:29.499335 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.496424 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9-systemd-units\") pod \"ovnkube-node-qv9kr\" (UID: \"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" Jul 21 02:09:29.499335 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.496450 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4bee3776-d7db-4be5-8e76-337ee2b89187-dbus\") pod \"global-pull-secret-syncer-ljdcc\" (UID: \"4bee3776-d7db-4be5-8e76-337ee2b89187\") " pod="kube-system/global-pull-secret-syncer-ljdcc" Jul 21 02:09:29.499335 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.496490 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9-systemd-units\") pod \"ovnkube-node-qv9kr\" (UID: \"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" Jul 21 02:09:29.499335 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.496521 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e0411e85-4230-45d9-9055-1e2edf4cd322-agent-certs\") pod \"konnectivity-agent-wj24l\" (UID: \"e0411e85-4230-45d9-9055-1e2edf4cd322\") " pod="kube-system/konnectivity-agent-wj24l" Jul 21 02:09:29.499335 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.496547 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4bee3776-d7db-4be5-8e76-337ee2b89187-dbus\") pod \"global-pull-secret-syncer-ljdcc\" (UID: \"4bee3776-d7db-4be5-8e76-337ee2b89187\") " pod="kube-system/global-pull-secret-syncer-ljdcc" Jul 21 02:09:29.499335 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.496585 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9-host-cni-netd\") pod \"ovnkube-node-qv9kr\" (UID: \"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" Jul 21 02:09:29.499335 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.497332 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91f30d91-79e3-44be-a679-ad1b6e873a99-tmp\") pod \"tuned-lgkbl\" (UID: \"91f30d91-79e3-44be-a679-ad1b6e873a99\") " pod="openshift-cluster-node-tuning-operator/tuned-lgkbl" Jul 21 02:09:29.499335 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.497376 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/91f30d91-79e3-44be-a679-ad1b6e873a99-etc-tuned\") pod \"tuned-lgkbl\" (UID: \"91f30d91-79e3-44be-a679-ad1b6e873a99\") " pod="openshift-cluster-node-tuning-operator/tuned-lgkbl" Jul 21 02:09:29.499335 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.497430 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9-ovn-node-metrics-cert\") pod \"ovnkube-node-qv9kr\" (UID: \"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" Jul 21 02:09:29.499335 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.498860 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e0411e85-4230-45d9-9055-1e2edf4cd322-agent-certs\") pod \"konnectivity-agent-wj24l\" (UID: \"e0411e85-4230-45d9-9055-1e2edf4cd322\") " pod="kube-system/konnectivity-agent-wj24l" Jul 21 02:09:29.505333 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:29.505312 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 02:09:29.505333 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:29.505335 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 02:09:29.505502 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:29.505363 2574 projected.go:196] Error preparing data for projected volume kube-api-access-ht26w for pod openshift-network-diagnostics/network-check-target-qqnx4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 02:09:29.505502 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:29.505443 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/80deee90-4b47-4cdf-bc87-f9242ad63cec-kube-api-access-ht26w podName:80deee90-4b47-4cdf-bc87-f9242ad63cec nodeName:}" failed. No retries permitted until 2026-07-21 02:09:30.005429055 +0000 UTC m=+4.024922940 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ht26w" (UniqueName: "kubernetes.io/projected/80deee90-4b47-4cdf-bc87-f9242ad63cec-kube-api-access-ht26w") pod "network-check-target-qqnx4" (UID: "80deee90-4b47-4cdf-bc87-f9242ad63cec") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 02:09:29.507978 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.507941 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8rtl\" (UniqueName: \"kubernetes.io/projected/48bafec4-325f-45e3-b658-5e5a9ff57629-kube-api-access-q8rtl\") pod \"aws-ebs-csi-driver-node-qsl9k\" (UID: \"48bafec4-325f-45e3-b658-5e5a9ff57629\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsl9k" Jul 21 02:09:29.508214 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.508188 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw5wm\" (UniqueName: \"kubernetes.io/projected/83fe72d8-5aa8-4232-9317-3e0d06bc9535-kube-api-access-sw5wm\") pod \"node-ca-b9rfh\" (UID: \"83fe72d8-5aa8-4232-9317-3e0d06bc9535\") " pod="openshift-image-registry/node-ca-b9rfh" Jul 21 02:09:29.510056 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.510007 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r474z\" (UniqueName: \"kubernetes.io/projected/c8e442c1-e98a-4af5-9274-5ab79f671aef-kube-api-access-r474z\") pod \"node-resolver-8d69j\" (UID: \"c8e442c1-e98a-4af5-9274-5ab79f671aef\") " pod="openshift-dns/node-resolver-8d69j" Jul 21 02:09:29.510139 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.510094 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9q8l\" (UniqueName: \"kubernetes.io/projected/df138c44-9b99-4291-9996-d10aee5562ff-kube-api-access-v9q8l\") pod \"iptables-alerter-75hhp\" (UID: \"df138c44-9b99-4291-9996-d10aee5562ff\") " pod="openshift-network-operator/iptables-alerter-75hhp" Jul 21 02:09:29.510372 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.510336 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl5wh\" (UniqueName: \"kubernetes.io/projected/91f30d91-79e3-44be-a679-ad1b6e873a99-kube-api-access-cl5wh\") pod \"tuned-lgkbl\" (UID: \"91f30d91-79e3-44be-a679-ad1b6e873a99\") " pod="openshift-cluster-node-tuning-operator/tuned-lgkbl" Jul 21 02:09:29.510469 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.510392 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7zpj\" (UniqueName: \"kubernetes.io/projected/bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9-kube-api-access-z7zpj\") pod \"ovnkube-node-qv9kr\" (UID: \"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" Jul 21 02:09:29.510715 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.510698 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6mrj\" (UniqueName: \"kubernetes.io/projected/51277326-7122-47d0-991c-4fe0c5aa0102-kube-api-access-f6mrj\") pod \"multus-lkmsc\" (UID: \"51277326-7122-47d0-991c-4fe0c5aa0102\") " pod="openshift-multus/multus-lkmsc" Jul 21 02:09:29.527526 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:29.527491 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03a5161441a8d59f0580132e9ff32a06.slice/crio-bd41da50a62ff6461bb9f01b8141ec066cd28d6910c22198c0469865061c1c70 WatchSource:0}: Error finding container bd41da50a62ff6461bb9f01b8141ec066cd28d6910c22198c0469865061c1c70: Status 404 returned error can't find the container with id bd41da50a62ff6461bb9f01b8141ec066cd28d6910c22198c0469865061c1c70 Jul 21 02:09:29.527965 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:29.527942 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96d07079af35c0ac51921ff8805ecb24.slice/crio-2501fd783639bc3159c458028b609b447a4368ff922031be436cfd0fa78137c8 WatchSource:0}: Error finding container 2501fd783639bc3159c458028b609b447a4368ff922031be436cfd0fa78137c8: Status 404 returned error can't find the container with id 2501fd783639bc3159c458028b609b447a4368ff922031be436cfd0fa78137c8 Jul 21 02:09:29.533831 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.533813 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 02:09:29.535789 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.535770 2574 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 02:09:29.613210 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.613113 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-x2w8b" Jul 21 02:09:29.623488 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:29.623454 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3504ec09_d4b6_41ce_b77d_d6c73b437a33.slice/crio-f7e53719188f9df39ce87c7093192d301772700b6e0e1998facb8e0e1b1a3915 WatchSource:0}: Error finding container f7e53719188f9df39ce87c7093192d301772700b6e0e1998facb8e0e1b1a3915: Status 404 returned error can't find the container with id f7e53719188f9df39ce87c7093192d301772700b6e0e1998facb8e0e1b1a3915 Jul 21 02:09:29.628242 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.628208 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-75hhp" Jul 21 02:09:29.636302 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.636262 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsl9k" Jul 21 02:09:29.636909 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:29.636872 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf138c44_9b99_4291_9996_d10aee5562ff.slice/crio-79a347ec53fc54d801bcb8cd1e77dfebf0989f437569bd67856c156e7ac70f56 WatchSource:0}: Error finding container 79a347ec53fc54d801bcb8cd1e77dfebf0989f437569bd67856c156e7ac70f56: Status 404 returned error can't find the container with id 79a347ec53fc54d801bcb8cd1e77dfebf0989f437569bd67856c156e7ac70f56 Jul 21 02:09:29.638503 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.638485 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lgkbl" Jul 21 02:09:29.645805 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:29.645778 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48bafec4_325f_45e3_b658_5e5a9ff57629.slice/crio-4306f50884c9eee572e8ae022338fa68e059c66cb6378e7ae865ed60c07f2821 WatchSource:0}: Error finding container 4306f50884c9eee572e8ae022338fa68e059c66cb6378e7ae865ed60c07f2821: Status 404 returned error can't find the container with id 4306f50884c9eee572e8ae022338fa68e059c66cb6378e7ae865ed60c07f2821 Jul 21 02:09:29.647740 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:29.647717 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91f30d91_79e3_44be_a679_ad1b6e873a99.slice/crio-a22ea0336eae5b00509389a5272b38905907adcc2612abf859f9692498caf743 WatchSource:0}: Error finding container a22ea0336eae5b00509389a5272b38905907adcc2612abf859f9692498caf743: Status 404 returned error can't find the container with id a22ea0336eae5b00509389a5272b38905907adcc2612abf859f9692498caf743 Jul 21 02:09:29.671656 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.671617 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8d69j" Jul 21 02:09:29.681424 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:29.681396 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8e442c1_e98a_4af5_9274_5ab79f671aef.slice/crio-4e927833d613715b400a2661db4193f96ce43137c714afe44cc7b0bbafd2b081 WatchSource:0}: Error finding container 4e927833d613715b400a2661db4193f96ce43137c714afe44cc7b0bbafd2b081: Status 404 returned error can't find the container with id 4e927833d613715b400a2661db4193f96ce43137c714afe44cc7b0bbafd2b081 Jul 21 02:09:29.682378 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.682325 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-b9rfh" Jul 21 02:09:29.690726 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:29.690701 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83fe72d8_5aa8_4232_9317_3e0d06bc9535.slice/crio-f7e3d261055107eb6f35af53bba670f2171dbb8c9fea63dcb829645682c41fc6 WatchSource:0}: Error finding container f7e3d261055107eb6f35af53bba670f2171dbb8c9fea63dcb829645682c41fc6: Status 404 returned error can't find the container with id f7e3d261055107eb6f35af53bba670f2171dbb8c9fea63dcb829645682c41fc6 Jul 21 02:09:29.700541 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.700519 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lkmsc" Jul 21 02:09:29.708860 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:29.708835 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51277326_7122_47d0_991c_4fe0c5aa0102.slice/crio-30b987e1809954cf3a054ac5208dc8fdd17e796f5214d1800a89746de1021e8e WatchSource:0}: Error finding container 30b987e1809954cf3a054ac5208dc8fdd17e796f5214d1800a89746de1021e8e: Status 404 returned error can't find the container with id 30b987e1809954cf3a054ac5208dc8fdd17e796f5214d1800a89746de1021e8e Jul 21 02:09:29.722359 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.722335 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" Jul 21 02:09:29.728459 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.728438 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wj24l" Jul 21 02:09:29.731054 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:29.731032 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf3f6ce0_b6b7_4896_b7b5_5c103e8f36b9.slice/crio-72ef7a01ca4562a53457986e7de870f1e6b42ebcea404dd6bfbb32baa512bc94 WatchSource:0}: Error finding container 72ef7a01ca4562a53457986e7de870f1e6b42ebcea404dd6bfbb32baa512bc94: Status 404 returned error can't find the container with id 72ef7a01ca4562a53457986e7de870f1e6b42ebcea404dd6bfbb32baa512bc94 Jul 21 02:09:29.737207 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:09:29.737185 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0411e85_4230_45d9_9055_1e2edf4cd322.slice/crio-a6f5de0a335441743f95d6bd9529f39e612977882e13201c8452c8f9eb7cc5c7 WatchSource:0}: Error finding container a6f5de0a335441743f95d6bd9529f39e612977882e13201c8452c8f9eb7cc5c7: Status 404 returned error can't find the container with id a6f5de0a335441743f95d6bd9529f39e612977882e13201c8452c8f9eb7cc5c7 Jul 21 02:09:29.898943 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:29.898860 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03e524b0-40bd-46b4-ac8f-eb346e94a7dc-metrics-certs\") pod \"network-metrics-daemon-8v67q\" (UID: \"03e524b0-40bd-46b4-ac8f-eb346e94a7dc\") " pod="openshift-multus/network-metrics-daemon-8v67q" Jul 21 02:09:29.899113 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:29.899065 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 02:09:29.899245 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:29.899156 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03e524b0-40bd-46b4-ac8f-eb346e94a7dc-metrics-certs podName:03e524b0-40bd-46b4-ac8f-eb346e94a7dc nodeName:}" failed. No retries permitted until 2026-07-21 02:09:30.899131785 +0000 UTC m=+4.918625691 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03e524b0-40bd-46b4-ac8f-eb346e94a7dc-metrics-certs") pod "network-metrics-daemon-8v67q" (UID: "03e524b0-40bd-46b4-ac8f-eb346e94a7dc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 02:09:30.000193 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:30.000151 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4bee3776-d7db-4be5-8e76-337ee2b89187-original-pull-secret\") pod \"global-pull-secret-syncer-ljdcc\" (UID: \"4bee3776-d7db-4be5-8e76-337ee2b89187\") " pod="kube-system/global-pull-secret-syncer-ljdcc" Jul 21 02:09:30.000403 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:30.000289 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 02:09:30.000403 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:30.000387 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bee3776-d7db-4be5-8e76-337ee2b89187-original-pull-secret podName:4bee3776-d7db-4be5-8e76-337ee2b89187 nodeName:}" failed. No retries permitted until 2026-07-21 02:09:31.000366056 +0000 UTC m=+5.019859956 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4bee3776-d7db-4be5-8e76-337ee2b89187-original-pull-secret") pod "global-pull-secret-syncer-ljdcc" (UID: "4bee3776-d7db-4be5-8e76-337ee2b89187") : object "kube-system"/"original-pull-secret" not registered Jul 21 02:09:30.101014 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:30.100968 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht26w\" (UniqueName: \"kubernetes.io/projected/80deee90-4b47-4cdf-bc87-f9242ad63cec-kube-api-access-ht26w\") pod \"network-check-target-qqnx4\" (UID: \"80deee90-4b47-4cdf-bc87-f9242ad63cec\") " pod="openshift-network-diagnostics/network-check-target-qqnx4" Jul 21 02:09:30.101252 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:30.101209 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 02:09:30.101252 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:30.101236 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 02:09:30.101252 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:30.101250 2574 projected.go:196] Error preparing data for projected volume kube-api-access-ht26w for pod openshift-network-diagnostics/network-check-target-qqnx4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 02:09:30.101484 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:30.101334 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/80deee90-4b47-4cdf-bc87-f9242ad63cec-kube-api-access-ht26w podName:80deee90-4b47-4cdf-bc87-f9242ad63cec nodeName:}" failed. No retries permitted until 2026-07-21 02:09:31.101292466 +0000 UTC m=+5.120786357 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ht26w" (UniqueName: "kubernetes.io/projected/80deee90-4b47-4cdf-bc87-f9242ad63cec-kube-api-access-ht26w") pod "network-check-target-qqnx4" (UID: "80deee90-4b47-4cdf-bc87-f9242ad63cec") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 02:09:30.351986 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:30.351792 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-20 02:04:29 +0000 UTC" deadline="2027-12-16 05:22:25.694393439 +0000 UTC" Jul 21 02:09:30.351986 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:30.351859 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12315h12m55.342539292s" Jul 21 02:09:30.482121 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:30.482084 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8v67q" Jul 21 02:09:30.482294 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:30.482233 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8v67q" podUID="03e524b0-40bd-46b4-ac8f-eb346e94a7dc" Jul 21 02:09:30.516915 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:30.516844 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lkmsc" event={"ID":"51277326-7122-47d0-991c-4fe0c5aa0102","Type":"ContainerStarted","Data":"30b987e1809954cf3a054ac5208dc8fdd17e796f5214d1800a89746de1021e8e"} Jul 21 02:09:30.539810 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:30.539767 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-b9rfh" event={"ID":"83fe72d8-5aa8-4232-9317-3e0d06bc9535","Type":"ContainerStarted","Data":"f7e3d261055107eb6f35af53bba670f2171dbb8c9fea63dcb829645682c41fc6"} Jul 21 02:09:30.571311 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:30.571265 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsl9k" event={"ID":"48bafec4-325f-45e3-b658-5e5a9ff57629","Type":"ContainerStarted","Data":"4306f50884c9eee572e8ae022338fa68e059c66cb6378e7ae865ed60c07f2821"} Jul 21 02:09:30.577788 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:30.577751 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-37.ec2.internal" event={"ID":"96d07079af35c0ac51921ff8805ecb24","Type":"ContainerStarted","Data":"2501fd783639bc3159c458028b609b447a4368ff922031be436cfd0fa78137c8"} Jul 21 02:09:30.585624 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:30.585587 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wj24l" event={"ID":"e0411e85-4230-45d9-9055-1e2edf4cd322","Type":"ContainerStarted","Data":"a6f5de0a335441743f95d6bd9529f39e612977882e13201c8452c8f9eb7cc5c7"} Jul 21 02:09:30.605889 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:30.605759 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" event={"ID":"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9","Type":"ContainerStarted","Data":"72ef7a01ca4562a53457986e7de870f1e6b42ebcea404dd6bfbb32baa512bc94"} Jul 21 02:09:30.619754 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:30.619712 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8d69j" event={"ID":"c8e442c1-e98a-4af5-9274-5ab79f671aef","Type":"ContainerStarted","Data":"4e927833d613715b400a2661db4193f96ce43137c714afe44cc7b0bbafd2b081"} Jul 21 02:09:30.634982 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:30.634936 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lgkbl" event={"ID":"91f30d91-79e3-44be-a679-ad1b6e873a99","Type":"ContainerStarted","Data":"a22ea0336eae5b00509389a5272b38905907adcc2612abf859f9692498caf743"} Jul 21 02:09:30.676438 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:30.676291 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-75hhp" event={"ID":"df138c44-9b99-4291-9996-d10aee5562ff","Type":"ContainerStarted","Data":"79a347ec53fc54d801bcb8cd1e77dfebf0989f437569bd67856c156e7ac70f56"} Jul 21 02:09:30.699692 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:30.699650 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x2w8b" event={"ID":"3504ec09-d4b6-41ce-b77d-d6c73b437a33","Type":"ContainerStarted","Data":"f7e53719188f9df39ce87c7093192d301772700b6e0e1998facb8e0e1b1a3915"} Jul 21 02:09:30.722847 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:30.722806 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-37.ec2.internal" event={"ID":"03a5161441a8d59f0580132e9ff32a06","Type":"ContainerStarted","Data":"bd41da50a62ff6461bb9f01b8141ec066cd28d6910c22198c0469865061c1c70"} Jul 21 02:09:30.764953 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:30.764918 2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 02:09:30.909901 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:30.909807 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03e524b0-40bd-46b4-ac8f-eb346e94a7dc-metrics-certs\") pod \"network-metrics-daemon-8v67q\" (UID: \"03e524b0-40bd-46b4-ac8f-eb346e94a7dc\") " pod="openshift-multus/network-metrics-daemon-8v67q" Jul 21 02:09:30.910085 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:30.910052 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 02:09:30.910154 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:30.910128 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03e524b0-40bd-46b4-ac8f-eb346e94a7dc-metrics-certs podName:03e524b0-40bd-46b4-ac8f-eb346e94a7dc nodeName:}" failed. No retries permitted until 2026-07-21 02:09:32.910106312 +0000 UTC m=+6.929600199 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03e524b0-40bd-46b4-ac8f-eb346e94a7dc-metrics-certs") pod "network-metrics-daemon-8v67q" (UID: "03e524b0-40bd-46b4-ac8f-eb346e94a7dc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 02:09:31.011394 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:31.011148 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4bee3776-d7db-4be5-8e76-337ee2b89187-original-pull-secret\") pod \"global-pull-secret-syncer-ljdcc\" (UID: \"4bee3776-d7db-4be5-8e76-337ee2b89187\") " pod="kube-system/global-pull-secret-syncer-ljdcc" Jul 21 02:09:31.011394 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:31.011369 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 02:09:31.011641 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:31.011450 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bee3776-d7db-4be5-8e76-337ee2b89187-original-pull-secret podName:4bee3776-d7db-4be5-8e76-337ee2b89187 nodeName:}" failed. No retries permitted until 2026-07-21 02:09:33.011426258 +0000 UTC m=+7.030920143 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4bee3776-d7db-4be5-8e76-337ee2b89187-original-pull-secret") pod "global-pull-secret-syncer-ljdcc" (UID: "4bee3776-d7db-4be5-8e76-337ee2b89187") : object "kube-system"/"original-pull-secret" not registered Jul 21 02:09:31.112185 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:31.111737 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht26w\" (UniqueName: \"kubernetes.io/projected/80deee90-4b47-4cdf-bc87-f9242ad63cec-kube-api-access-ht26w\") pod \"network-check-target-qqnx4\" (UID: \"80deee90-4b47-4cdf-bc87-f9242ad63cec\") " pod="openshift-network-diagnostics/network-check-target-qqnx4" Jul 21 02:09:31.112185 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:31.111973 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 02:09:31.112185 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:31.111997 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 02:09:31.112185 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:31.112010 2574 projected.go:196] Error preparing data for projected volume kube-api-access-ht26w for pod openshift-network-diagnostics/network-check-target-qqnx4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 02:09:31.112185 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:31.112074 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/80deee90-4b47-4cdf-bc87-f9242ad63cec-kube-api-access-ht26w podName:80deee90-4b47-4cdf-bc87-f9242ad63cec nodeName:}" failed. No retries permitted until 2026-07-21 02:09:33.112052961 +0000 UTC m=+7.131546861 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ht26w" (UniqueName: "kubernetes.io/projected/80deee90-4b47-4cdf-bc87-f9242ad63cec-kube-api-access-ht26w") pod "network-check-target-qqnx4" (UID: "80deee90-4b47-4cdf-bc87-f9242ad63cec") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 02:09:31.352220 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:31.352122 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-20 02:04:29 +0000 UTC" deadline="2028-03-01 04:46:05.476855755 +0000 UTC" Jul 21 02:09:31.352220 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:31.352171 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14138h36m34.12468999s" Jul 21 02:09:31.478902 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:31.478863 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ljdcc" Jul 21 02:09:31.479106 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:31.479029 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ljdcc" podUID="4bee3776-d7db-4be5-8e76-337ee2b89187" Jul 21 02:09:31.479516 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:31.479491 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqnx4" Jul 21 02:09:31.479634 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:31.479603 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qqnx4" podUID="80deee90-4b47-4cdf-bc87-f9242ad63cec" Jul 21 02:09:32.181527 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:32.181484 2574 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 02:09:32.479183 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:32.479096 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8v67q" Jul 21 02:09:32.479700 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:32.479319 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8v67q" podUID="03e524b0-40bd-46b4-ac8f-eb346e94a7dc" Jul 21 02:09:32.928971 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:32.928889 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03e524b0-40bd-46b4-ac8f-eb346e94a7dc-metrics-certs\") pod \"network-metrics-daemon-8v67q\" (UID: \"03e524b0-40bd-46b4-ac8f-eb346e94a7dc\") " pod="openshift-multus/network-metrics-daemon-8v67q" Jul 21 02:09:32.929116 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:32.929073 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 02:09:32.929179 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:32.929141 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03e524b0-40bd-46b4-ac8f-eb346e94a7dc-metrics-certs podName:03e524b0-40bd-46b4-ac8f-eb346e94a7dc nodeName:}" failed. No retries permitted until 2026-07-21 02:09:36.929121509 +0000 UTC m=+10.948615392 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03e524b0-40bd-46b4-ac8f-eb346e94a7dc-metrics-certs") pod "network-metrics-daemon-8v67q" (UID: "03e524b0-40bd-46b4-ac8f-eb346e94a7dc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 02:09:33.030091 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:33.030056 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4bee3776-d7db-4be5-8e76-337ee2b89187-original-pull-secret\") pod \"global-pull-secret-syncer-ljdcc\" (UID: \"4bee3776-d7db-4be5-8e76-337ee2b89187\") " pod="kube-system/global-pull-secret-syncer-ljdcc" Jul 21 02:09:33.030327 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:33.030235 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 02:09:33.030327 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:33.030314 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bee3776-d7db-4be5-8e76-337ee2b89187-original-pull-secret podName:4bee3776-d7db-4be5-8e76-337ee2b89187 nodeName:}" failed. No retries permitted until 2026-07-21 02:09:37.030297049 +0000 UTC m=+11.049790949 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4bee3776-d7db-4be5-8e76-337ee2b89187-original-pull-secret") pod "global-pull-secret-syncer-ljdcc" (UID: "4bee3776-d7db-4be5-8e76-337ee2b89187") : object "kube-system"/"original-pull-secret" not registered Jul 21 02:09:33.131326 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:33.131286 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht26w\" (UniqueName: \"kubernetes.io/projected/80deee90-4b47-4cdf-bc87-f9242ad63cec-kube-api-access-ht26w\") pod \"network-check-target-qqnx4\" (UID: \"80deee90-4b47-4cdf-bc87-f9242ad63cec\") " pod="openshift-network-diagnostics/network-check-target-qqnx4" Jul 21 02:09:33.131557 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:33.131537 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 02:09:33.131630 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:33.131569 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 02:09:33.131630 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:33.131582 2574 projected.go:196] Error preparing data for projected volume kube-api-access-ht26w for pod openshift-network-diagnostics/network-check-target-qqnx4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 02:09:33.131745 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:33.131640 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/80deee90-4b47-4cdf-bc87-f9242ad63cec-kube-api-access-ht26w podName:80deee90-4b47-4cdf-bc87-f9242ad63cec nodeName:}" failed. No retries permitted until 2026-07-21 02:09:37.131622323 +0000 UTC m=+11.151116229 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ht26w" (UniqueName: "kubernetes.io/projected/80deee90-4b47-4cdf-bc87-f9242ad63cec-kube-api-access-ht26w") pod "network-check-target-qqnx4" (UID: "80deee90-4b47-4cdf-bc87-f9242ad63cec") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 02:09:33.479369 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:33.479334 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqnx4" Jul 21 02:09:33.479819 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:33.479462 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qqnx4" podUID="80deee90-4b47-4cdf-bc87-f9242ad63cec" Jul 21 02:09:33.479819 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:33.479334 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ljdcc" Jul 21 02:09:33.479937 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:33.479904 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ljdcc" podUID="4bee3776-d7db-4be5-8e76-337ee2b89187" Jul 21 02:09:34.479913 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:34.479737 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8v67q" Jul 21 02:09:34.480833 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:34.480479 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8v67q" podUID="03e524b0-40bd-46b4-ac8f-eb346e94a7dc" Jul 21 02:09:35.478819 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:35.478778 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqnx4" Jul 21 02:09:35.479006 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:35.478825 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ljdcc" Jul 21 02:09:35.479006 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:35.478909 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qqnx4" podUID="80deee90-4b47-4cdf-bc87-f9242ad63cec" Jul 21 02:09:35.479422 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:35.479389 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ljdcc" podUID="4bee3776-d7db-4be5-8e76-337ee2b89187" Jul 21 02:09:36.479181 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:36.479134 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8v67q" Jul 21 02:09:36.479655 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:36.479300 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8v67q" podUID="03e524b0-40bd-46b4-ac8f-eb346e94a7dc" Jul 21 02:09:36.967007 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:36.966919 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03e524b0-40bd-46b4-ac8f-eb346e94a7dc-metrics-certs\") pod \"network-metrics-daemon-8v67q\" (UID: \"03e524b0-40bd-46b4-ac8f-eb346e94a7dc\") " pod="openshift-multus/network-metrics-daemon-8v67q" Jul 21 02:09:36.967169 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:36.967087 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 02:09:36.967169 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:36.967144 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03e524b0-40bd-46b4-ac8f-eb346e94a7dc-metrics-certs podName:03e524b0-40bd-46b4-ac8f-eb346e94a7dc nodeName:}" failed. No retries permitted until 2026-07-21 02:09:44.967124675 +0000 UTC m=+18.986618558 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03e524b0-40bd-46b4-ac8f-eb346e94a7dc-metrics-certs") pod "network-metrics-daemon-8v67q" (UID: "03e524b0-40bd-46b4-ac8f-eb346e94a7dc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 02:09:37.067884 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:37.067789 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4bee3776-d7db-4be5-8e76-337ee2b89187-original-pull-secret\") pod \"global-pull-secret-syncer-ljdcc\" (UID: \"4bee3776-d7db-4be5-8e76-337ee2b89187\") " pod="kube-system/global-pull-secret-syncer-ljdcc" Jul 21 02:09:37.068090 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:37.067954 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 02:09:37.068090 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:37.068040 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bee3776-d7db-4be5-8e76-337ee2b89187-original-pull-secret podName:4bee3776-d7db-4be5-8e76-337ee2b89187 nodeName:}" failed. No retries permitted until 2026-07-21 02:09:45.068007643 +0000 UTC m=+19.087501546 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4bee3776-d7db-4be5-8e76-337ee2b89187-original-pull-secret") pod "global-pull-secret-syncer-ljdcc" (UID: "4bee3776-d7db-4be5-8e76-337ee2b89187") : object "kube-system"/"original-pull-secret" not registered Jul 21 02:09:37.168495 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:37.168459 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht26w\" (UniqueName: \"kubernetes.io/projected/80deee90-4b47-4cdf-bc87-f9242ad63cec-kube-api-access-ht26w\") pod \"network-check-target-qqnx4\" (UID: \"80deee90-4b47-4cdf-bc87-f9242ad63cec\") " pod="openshift-network-diagnostics/network-check-target-qqnx4" Jul 21 02:09:37.168748 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:37.168677 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 02:09:37.168748 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:37.168706 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 02:09:37.168748 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:37.168719 2574 projected.go:196] Error preparing data for projected volume kube-api-access-ht26w for pod openshift-network-diagnostics/network-check-target-qqnx4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 02:09:37.168947 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:37.168785 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/80deee90-4b47-4cdf-bc87-f9242ad63cec-kube-api-access-ht26w podName:80deee90-4b47-4cdf-bc87-f9242ad63cec nodeName:}" failed. No retries permitted until 2026-07-21 02:09:45.168767424 +0000 UTC m=+19.188261323 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ht26w" (UniqueName: "kubernetes.io/projected/80deee90-4b47-4cdf-bc87-f9242ad63cec-kube-api-access-ht26w") pod "network-check-target-qqnx4" (UID: "80deee90-4b47-4cdf-bc87-f9242ad63cec") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 02:09:37.479210 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:37.479171 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqnx4" Jul 21 02:09:37.479210 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:37.479209 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ljdcc" Jul 21 02:09:37.479745 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:37.479352 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qqnx4" podUID="80deee90-4b47-4cdf-bc87-f9242ad63cec" Jul 21 02:09:37.479745 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:37.479439 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ljdcc" podUID="4bee3776-d7db-4be5-8e76-337ee2b89187" Jul 21 02:09:38.480410 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:38.480373 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8v67q" Jul 21 02:09:38.480800 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:38.480499 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8v67q" podUID="03e524b0-40bd-46b4-ac8f-eb346e94a7dc" Jul 21 02:09:39.478902 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:39.478811 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ljdcc" Jul 21 02:09:39.479057 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:39.478934 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ljdcc" podUID="4bee3776-d7db-4be5-8e76-337ee2b89187" Jul 21 02:09:39.479300 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:39.479279 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqnx4" Jul 21 02:09:39.479386 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:39.479363 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qqnx4" podUID="80deee90-4b47-4cdf-bc87-f9242ad63cec" Jul 21 02:09:40.479355 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:40.479318 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8v67q" Jul 21 02:09:40.479796 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:40.479457 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8v67q" podUID="03e524b0-40bd-46b4-ac8f-eb346e94a7dc" Jul 21 02:09:41.478766 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:41.478738 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqnx4" Jul 21 02:09:41.478866 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:41.478748 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ljdcc" Jul 21 02:09:41.478866 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:41.478835 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qqnx4" podUID="80deee90-4b47-4cdf-bc87-f9242ad63cec" Jul 21 02:09:41.478949 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:41.478908 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ljdcc" podUID="4bee3776-d7db-4be5-8e76-337ee2b89187" Jul 21 02:09:42.478546 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:42.478506 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8v67q" Jul 21 02:09:42.478991 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:42.478658 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8v67q" podUID="03e524b0-40bd-46b4-ac8f-eb346e94a7dc" Jul 21 02:09:43.478618 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:43.478586 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqnx4" Jul 21 02:09:43.478618 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:43.478610 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ljdcc" Jul 21 02:09:43.479147 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:43.478713 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qqnx4" podUID="80deee90-4b47-4cdf-bc87-f9242ad63cec" Jul 21 02:09:43.479147 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:43.478845 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ljdcc" podUID="4bee3776-d7db-4be5-8e76-337ee2b89187" Jul 21 02:09:44.479408 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:44.479259 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8v67q" Jul 21 02:09:44.479881 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:44.479444 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8v67q" podUID="03e524b0-40bd-46b4-ac8f-eb346e94a7dc" Jul 21 02:09:45.024950 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:45.024907 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03e524b0-40bd-46b4-ac8f-eb346e94a7dc-metrics-certs\") pod \"network-metrics-daemon-8v67q\" (UID: \"03e524b0-40bd-46b4-ac8f-eb346e94a7dc\") " pod="openshift-multus/network-metrics-daemon-8v67q" Jul 21 02:09:45.025191 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:45.025049 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 02:09:45.025191 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:45.025120 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03e524b0-40bd-46b4-ac8f-eb346e94a7dc-metrics-certs podName:03e524b0-40bd-46b4-ac8f-eb346e94a7dc nodeName:}" failed. No retries permitted until 2026-07-21 02:10:01.025102634 +0000 UTC m=+35.044596538 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03e524b0-40bd-46b4-ac8f-eb346e94a7dc-metrics-certs") pod "network-metrics-daemon-8v67q" (UID: "03e524b0-40bd-46b4-ac8f-eb346e94a7dc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 02:09:45.125960 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:45.125911 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4bee3776-d7db-4be5-8e76-337ee2b89187-original-pull-secret\") pod \"global-pull-secret-syncer-ljdcc\" (UID: \"4bee3776-d7db-4be5-8e76-337ee2b89187\") " pod="kube-system/global-pull-secret-syncer-ljdcc" Jul 21 02:09:45.126135 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:45.126066 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 02:09:45.126135 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:45.126134 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bee3776-d7db-4be5-8e76-337ee2b89187-original-pull-secret podName:4bee3776-d7db-4be5-8e76-337ee2b89187 nodeName:}" failed. No retries permitted until 2026-07-21 02:10:01.126120007 +0000 UTC m=+35.145613894 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4bee3776-d7db-4be5-8e76-337ee2b89187-original-pull-secret") pod "global-pull-secret-syncer-ljdcc" (UID: "4bee3776-d7db-4be5-8e76-337ee2b89187") : object "kube-system"/"original-pull-secret" not registered Jul 21 02:09:45.227324 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:45.227277 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht26w\" (UniqueName: \"kubernetes.io/projected/80deee90-4b47-4cdf-bc87-f9242ad63cec-kube-api-access-ht26w\") pod \"network-check-target-qqnx4\" (UID: \"80deee90-4b47-4cdf-bc87-f9242ad63cec\") " pod="openshift-network-diagnostics/network-check-target-qqnx4" Jul 21 02:09:45.227473 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:45.227410 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 02:09:45.227473 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:45.227426 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 02:09:45.227473 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:45.227436 2574 projected.go:196] Error preparing data for projected volume kube-api-access-ht26w for pod openshift-network-diagnostics/network-check-target-qqnx4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 02:09:45.227613 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:45.227487 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/80deee90-4b47-4cdf-bc87-f9242ad63cec-kube-api-access-ht26w podName:80deee90-4b47-4cdf-bc87-f9242ad63cec nodeName:}" failed. No retries permitted until 2026-07-21 02:10:01.227473287 +0000 UTC m=+35.246967189 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ht26w" (UniqueName: "kubernetes.io/projected/80deee90-4b47-4cdf-bc87-f9242ad63cec-kube-api-access-ht26w") pod "network-check-target-qqnx4" (UID: "80deee90-4b47-4cdf-bc87-f9242ad63cec") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 02:09:45.478459 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:45.478416 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqnx4" Jul 21 02:09:45.478637 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:45.478549 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qqnx4" podUID="80deee90-4b47-4cdf-bc87-f9242ad63cec" Jul 21 02:09:45.478637 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:45.478599 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ljdcc" Jul 21 02:09:45.478748 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:45.478727 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ljdcc" podUID="4bee3776-d7db-4be5-8e76-337ee2b89187" Jul 21 02:09:46.479317 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:46.479286 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8v67q" Jul 21 02:09:46.479718 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:46.479419 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8v67q" podUID="03e524b0-40bd-46b4-ac8f-eb346e94a7dc" Jul 21 02:09:47.478885 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:47.478843 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqnx4" Jul 21 02:09:47.479082 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:47.478893 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ljdcc" Jul 21 02:09:47.479082 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:47.478975 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qqnx4" podUID="80deee90-4b47-4cdf-bc87-f9242ad63cec" Jul 21 02:09:47.479082 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:47.479068 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ljdcc" podUID="4bee3776-d7db-4be5-8e76-337ee2b89187" Jul 21 02:09:48.479337 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:48.479295 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8v67q" Jul 21 02:09:48.479723 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:48.479465 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8v67q" podUID="03e524b0-40bd-46b4-ac8f-eb346e94a7dc" Jul 21 02:09:49.479395 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:49.479351 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ljdcc" Jul 21 02:09:49.479892 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:49.479405 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqnx4" Jul 21 02:09:49.479892 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:49.479492 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ljdcc" podUID="4bee3776-d7db-4be5-8e76-337ee2b89187" Jul 21 02:09:49.479892 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:49.479653 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qqnx4" podUID="80deee90-4b47-4cdf-bc87-f9242ad63cec" Jul 21 02:09:50.479584 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:50.479288 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8v67q" Jul 21 02:09:50.480083 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:50.479693 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8v67q" podUID="03e524b0-40bd-46b4-ac8f-eb346e94a7dc" Jul 21 02:09:50.794089 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:50.794053 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsl9k" event={"ID":"48bafec4-325f-45e3-b658-5e5a9ff57629","Type":"ContainerStarted","Data":"9f261cadf9bbbfad802f0fbac674b48a5909195574b4190293698195c1a4ba3a"} Jul 21 02:09:50.798714 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:50.798688 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wj24l" event={"ID":"e0411e85-4230-45d9-9055-1e2edf4cd322","Type":"ContainerStarted","Data":"abeaab2b1d37924e112fed460fdf9057ec58148746cf376f63db26ae3ceded9c"} Jul 21 02:09:50.802513 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:50.802489 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qv9kr_bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9/ovn-acl-logging/0.log" Jul 21 02:09:50.803172 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:50.803005 2574 generic.go:356] "Generic (PLEG): container finished" podID="bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9" containerID="9be6fd2dc381e6565b6a37bdc648fb2ac436799d5b9a4ab901550a712713c205" exitCode=1 Jul 21 02:09:50.803172 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:50.803087 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" event={"ID":"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9","Type":"ContainerStarted","Data":"f6848029a18ba507840cb431ef467472452aefd787dfd12fc873461f1b486838"} Jul 21 02:09:50.803350 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:50.803186 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" event={"ID":"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9","Type":"ContainerStarted","Data":"a41a4ea98fabbe63a8b6b9d4d9a60844de6930bf2769396e70919bc1d9c254ce"} Jul 21 02:09:50.803350 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:50.803239 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" event={"ID":"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9","Type":"ContainerStarted","Data":"0a0f6b85bde336cb5b17599b0a3f261f83f11a5d356dd1425117c42ca0e0cd6c"} Jul 21 02:09:50.803350 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:50.803310 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" event={"ID":"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9","Type":"ContainerStarted","Data":"3a7973d08365bb0563843a1251fcb3cd411030edc6304c7c637a30fbf875fdb7"} Jul 21 02:09:50.803350 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:50.803342 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" event={"ID":"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9","Type":"ContainerDied","Data":"9be6fd2dc381e6565b6a37bdc648fb2ac436799d5b9a4ab901550a712713c205"} Jul 21 02:09:50.803559 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:50.803373 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" event={"ID":"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9","Type":"ContainerStarted","Data":"c123a5899cc1d06041ea6f8514c4879ef588cde61d30288ad8702a5dcf33cf95"} Jul 21 02:09:50.806778 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:50.806711 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lgkbl" event={"ID":"91f30d91-79e3-44be-a679-ad1b6e873a99","Type":"ContainerStarted","Data":"57440cfe4fc258c9e12bffff5c1160f3610659319be41a25af34dfbd7fa83be0"} Jul 21 02:09:50.813717 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:50.813637 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-37.ec2.internal" event={"ID":"03a5161441a8d59f0580132e9ff32a06","Type":"ContainerStarted","Data":"ab50555135a7103d6a558b1c0707519355b03ea5cf3d54eba58c0992d4f0ba0b"} Jul 21 02:09:50.815552 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:50.815527 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lkmsc" event={"ID":"51277326-7122-47d0-991c-4fe0c5aa0102","Type":"ContainerStarted","Data":"218cd31c65ff107a7ffb99d622814415912b3ac113ac2e585fdcd7ce490e475e"} Jul 21 02:09:50.821555 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:50.821491 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-wj24l" podStartSLOduration=7.235810682 podStartE2EDuration="22.821474109s" totalImagesPullingTime="15.585663427s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 02:09:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:09:50.820607363 +0000 UTC m=+24.840101288" watchObservedRunningTime="2026-07-21 02:09:50.821474109 +0000 UTC m=+24.840968016" Jul 21 02:09:50.861110 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:50.861065 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-lgkbl" podStartSLOduration=2.532539298 podStartE2EDuration="22.861049827s" totalImagesPullingTime="20.328510529s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 02:09:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:09:50.860785387 +0000 UTC m=+24.880279292" watchObservedRunningTime="2026-07-21 02:09:50.861049827 +0000 UTC m=+24.880543731" Jul 21 02:09:50.880483 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:50.880434 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-lkmsc" podStartSLOduration=2.233358972 podStartE2EDuration="22.880417745s" totalImagesPullingTime="20.647058773s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 02:09:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:09:50.880064145 +0000 UTC m=+24.899558050" watchObservedRunningTime="2026-07-21 02:09:50.880417745 +0000 UTC m=+24.899911649" Jul 21 02:09:50.895161 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:50.895117 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-37.ec2.internal" podStartSLOduration=21.895107412 podStartE2EDuration="21.895107412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 02:09:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:09:50.89469873 +0000 UTC m=+24.914192634" watchObservedRunningTime="2026-07-21 02:09:50.895107412 +0000 UTC m=+24.914601317" Jul 21 02:09:51.114850 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:51.114774 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-wj24l" Jul 21 02:09:51.115510 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:51.115493 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-wj24l" Jul 21 02:09:51.478740 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:51.478423 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqnx4" Jul 21 02:09:51.478916 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:51.478812 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qqnx4" podUID="80deee90-4b47-4cdf-bc87-f9242ad63cec" Jul 21 02:09:51.478916 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:51.478476 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ljdcc" Jul 21 02:09:51.479009 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:51.478951 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ljdcc" podUID="4bee3776-d7db-4be5-8e76-337ee2b89187" Jul 21 02:09:51.819215 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:51.819132 2574 generic.go:356] "Generic (PLEG): container finished" podID="96d07079af35c0ac51921ff8805ecb24" containerID="22cad03c153c2ce7365c7e99ba6037b3134b1f30240efd52463039e9794a6a88" exitCode=0 Jul 21 02:09:51.819926 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:51.819221 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-37.ec2.internal" event={"ID":"96d07079af35c0ac51921ff8805ecb24","Type":"ContainerDied","Data":"22cad03c153c2ce7365c7e99ba6037b3134b1f30240efd52463039e9794a6a88"} Jul 21 02:09:51.820692 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:51.820673 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8d69j" event={"ID":"c8e442c1-e98a-4af5-9274-5ab79f671aef","Type":"ContainerStarted","Data":"aafd887fbb86eeb333a115856be4894205fb00271f8cc2502c2617996e7f2356"} Jul 21 02:09:51.821998 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:51.821979 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-75hhp" event={"ID":"df138c44-9b99-4291-9996-d10aee5562ff","Type":"ContainerStarted","Data":"d8ed9d5be5d358cfb3f74dde1225f5bfa877bf634371bee672dec20a0d071cd7"} Jul 21 02:09:51.823362 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:51.823334 2574 generic.go:356] "Generic (PLEG): container finished" podID="3504ec09-d4b6-41ce-b77d-d6c73b437a33" containerID="0e91e00634267f0a8a6a9daca69ca258587567dfe4ea8de8610d83250c3d98ab" exitCode=0 Jul 21 02:09:51.823448 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:51.823405 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x2w8b" event={"ID":"3504ec09-d4b6-41ce-b77d-d6c73b437a33","Type":"ContainerDied","Data":"0e91e00634267f0a8a6a9daca69ca258587567dfe4ea8de8610d83250c3d98ab"} Jul 21 02:09:51.824688 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:51.824668 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-b9rfh" event={"ID":"83fe72d8-5aa8-4232-9317-3e0d06bc9535","Type":"ContainerStarted","Data":"a559b25a195bb3926e8f756ccc3d9d0bf5f3d1442c9dd8d7490545f4504ebf2e"} Jul 21 02:09:51.825155 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:51.825138 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-wj24l" Jul 21 02:09:51.825459 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:51.825444 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wj24l" Jul 21 02:09:51.854624 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:51.854574 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-75hhp" podStartSLOduration=3.537778793 podStartE2EDuration="23.854558832s" totalImagesPullingTime="20.316780039s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 02:09:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:09:51.854008902 +0000 UTC m=+25.873502807" watchObservedRunningTime="2026-07-21 02:09:51.854558832 +0000 UTC m=+25.874052736" Jul 21 02:09:51.872151 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:51.872076 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-8d69j" podStartSLOduration=3.576068289 podStartE2EDuration="23.872059104s" totalImagesPullingTime="20.295990815s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 02:09:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:09:51.871373427 +0000 UTC m=+25.890867355" watchObservedRunningTime="2026-07-21 02:09:51.872059104 +0000 UTC m=+25.891553011" Jul 21 02:09:51.909753 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:51.909709 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-b9rfh" podStartSLOduration=3.625841626 podStartE2EDuration="23.909695607s" totalImagesPullingTime="20.283853981s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 02:09:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:09:51.885789717 +0000 UTC m=+25.905283633" watchObservedRunningTime="2026-07-21 02:09:51.909695607 +0000 UTC m=+25.929189548" Jul 21 02:09:52.012610 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:52.012581 2574 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 21 02:09:52.366929 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:52.366829 2574 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-21T02:09:52.012602116Z","UUID":"98bc0566-7b75-4ffe-9c71-8810935497aa","Handler":null,"Name":"","Endpoint":""} Jul 21 02:09:52.368423 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:52.368400 2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 21 02:09:52.368554 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:52.368431 2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 21 02:09:52.478800 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:52.478763 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8v67q" Jul 21 02:09:52.478994 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:52.478947 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8v67q" podUID="03e524b0-40bd-46b4-ac8f-eb346e94a7dc" Jul 21 02:09:52.829967 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:52.829916 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsl9k" event={"ID":"48bafec4-325f-45e3-b658-5e5a9ff57629","Type":"ContainerStarted","Data":"5b8eed567f6bc48d844524538f359634eb78104cd915fce71a9519b3b97f322c"} Jul 21 02:09:52.831983 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:52.831948 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-37.ec2.internal" event={"ID":"96d07079af35c0ac51921ff8805ecb24","Type":"ContainerStarted","Data":"921bcb115a1d76fb5fff8db2877649ba388814794aad771c1d55fe8ed24dc300"} Jul 21 02:09:52.849899 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:52.849855 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-37.ec2.internal" podStartSLOduration=23.849839518 podStartE2EDuration="23.849839518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 02:09:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:09:52.849561137 +0000 UTC m=+26.869055055" watchObservedRunningTime="2026-07-21 02:09:52.849839518 +0000 UTC m=+26.869333422" Jul 21 02:09:53.479350 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:53.479254 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqnx4" Jul 21 02:09:53.479554 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:53.479384 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qqnx4" podUID="80deee90-4b47-4cdf-bc87-f9242ad63cec" Jul 21 02:09:53.479648 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:53.479258 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ljdcc" Jul 21 02:09:53.479775 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:53.479756 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ljdcc" podUID="4bee3776-d7db-4be5-8e76-337ee2b89187" Jul 21 02:09:53.837216 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:53.837142 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsl9k" event={"ID":"48bafec4-325f-45e3-b658-5e5a9ff57629","Type":"ContainerStarted","Data":"70e19c4bbecb7d91d6233dacc332e1dad3d257593ffbc0b505c8ccc2b6bc0d41"} Jul 21 02:09:53.840679 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:53.840653 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qv9kr_bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9/ovn-acl-logging/0.log" Jul 21 02:09:53.841122 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:53.841101 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" event={"ID":"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9","Type":"ContainerStarted","Data":"494ce6482fac5b05eac7d4c9b508f6cde04989754f5cc15a9a09004ff65cf702"} Jul 21 02:09:54.479499 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:54.479470 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8v67q" Jul 21 02:09:54.479635 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:54.479587 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8v67q" podUID="03e524b0-40bd-46b4-ac8f-eb346e94a7dc" Jul 21 02:09:55.478919 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:55.478834 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ljdcc" Jul 21 02:09:55.479380 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:55.478971 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ljdcc" podUID="4bee3776-d7db-4be5-8e76-337ee2b89187" Jul 21 02:09:55.479380 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:55.479022 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqnx4" Jul 21 02:09:55.479380 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:55.479114 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qqnx4" podUID="80deee90-4b47-4cdf-bc87-f9242ad63cec" Jul 21 02:09:56.479335 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:56.479154 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8v67q" Jul 21 02:09:56.479897 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:56.479435 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8v67q" podUID="03e524b0-40bd-46b4-ac8f-eb346e94a7dc" Jul 21 02:09:56.850424 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:56.850401 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qv9kr_bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9/ovn-acl-logging/0.log" Jul 21 02:09:56.850764 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:56.850740 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" event={"ID":"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9","Type":"ContainerStarted","Data":"0424e8bf16807afcca0183ff8ebac9af13c25d179115123821d23290f2c9e17f"} Jul 21 02:09:56.850983 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:56.850968 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" Jul 21 02:09:56.851183 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:56.851162 2574 scope.go:117] "RemoveContainer" containerID="9be6fd2dc381e6565b6a37bdc648fb2ac436799d5b9a4ab901550a712713c205" Jul 21 02:09:56.852630 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:56.852606 2574 generic.go:356] "Generic (PLEG): container finished" podID="3504ec09-d4b6-41ce-b77d-d6c73b437a33" containerID="01c3ad275d030502b096c1f987a6184ddce791e5503da35dbeb94e08e9f5c3b1" exitCode=0 Jul 21 02:09:56.852696 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:56.852659 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x2w8b" event={"ID":"3504ec09-d4b6-41ce-b77d-d6c73b437a33","Type":"ContainerDied","Data":"01c3ad275d030502b096c1f987a6184ddce791e5503da35dbeb94e08e9f5c3b1"} Jul 21 02:09:56.867757 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:56.867736 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" Jul 21 02:09:56.883936 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:56.883896 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsl9k" podStartSLOduration=6.314390129 podStartE2EDuration="28.883881606s" totalImagesPullingTime="22.569491477s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 02:09:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:09:53.860751997 +0000 UTC m=+27.880245903" watchObservedRunningTime="2026-07-21 02:09:56.883881606 +0000 UTC m=+30.903375511" Jul 21 02:09:57.479065 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:57.479031 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqnx4" Jul 21 02:09:57.479241 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:57.479130 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qqnx4" podUID="80deee90-4b47-4cdf-bc87-f9242ad63cec" Jul 21 02:09:57.479241 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:57.479161 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ljdcc" Jul 21 02:09:57.479241 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:57.479209 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ljdcc" podUID="4bee3776-d7db-4be5-8e76-337ee2b89187" Jul 21 02:09:57.862614 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:57.862420 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qv9kr_bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9/ovn-acl-logging/0.log" Jul 21 02:09:57.863226 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:57.862816 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" event={"ID":"bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9","Type":"ContainerStarted","Data":"f0c7f68103f3d89ea463b300fdbc435a1dce8b39a04a181ed5397cfd2e31f91d"} Jul 21 02:09:57.863464 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:57.863416 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" Jul 21 02:09:57.863615 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:57.863525 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" Jul 21 02:09:57.882380 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:57.882353 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" Jul 21 02:09:57.893021 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:57.892981 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" podStartSLOduration=9.596283477 podStartE2EDuration="29.892964292s" totalImagesPullingTime="20.296680815s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 02:09:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:09:57.891510441 +0000 UTC m=+31.911004365" watchObservedRunningTime="2026-07-21 02:09:57.892964292 +0000 UTC m=+31.912458220" Jul 21 02:09:58.113526 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:58.113296 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qqnx4"] Jul 21 02:09:58.113654 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:58.113550 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqnx4" Jul 21 02:09:58.113654 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:58.113636 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qqnx4" podUID="80deee90-4b47-4cdf-bc87-f9242ad63cec" Jul 21 02:09:58.116448 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:58.116419 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ljdcc"] Jul 21 02:09:58.116566 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:58.116510 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ljdcc" Jul 21 02:09:58.116637 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:58.116615 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ljdcc" podUID="4bee3776-d7db-4be5-8e76-337ee2b89187" Jul 21 02:09:58.117094 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:58.117072 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8v67q"] Jul 21 02:09:58.117181 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:58.117170 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8v67q" Jul 21 02:09:58.117300 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:58.117277 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8v67q" podUID="03e524b0-40bd-46b4-ac8f-eb346e94a7dc" Jul 21 02:09:58.867036 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:58.867003 2574 generic.go:356] "Generic (PLEG): container finished" podID="3504ec09-d4b6-41ce-b77d-d6c73b437a33" containerID="9b268ac991f86db5d51e04b2f6ec79cdca40ad5a2188000161c10aaa92ba35ea" exitCode=0 Jul 21 02:09:58.867470 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:58.867081 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x2w8b" event={"ID":"3504ec09-d4b6-41ce-b77d-d6c73b437a33","Type":"ContainerDied","Data":"9b268ac991f86db5d51e04b2f6ec79cdca40ad5a2188000161c10aaa92ba35ea"} Jul 21 02:09:59.478488 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:09:59.478416 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqnx4" Jul 21 02:09:59.478628 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:09:59.478539 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qqnx4" podUID="80deee90-4b47-4cdf-bc87-f9242ad63cec" Jul 21 02:10:00.478778 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:00.478701 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8v67q" Jul 21 02:10:00.478778 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:00.478743 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ljdcc" Jul 21 02:10:00.479161 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:10:00.478820 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8v67q" podUID="03e524b0-40bd-46b4-ac8f-eb346e94a7dc" Jul 21 02:10:00.479161 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:10:00.478958 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ljdcc" podUID="4bee3776-d7db-4be5-8e76-337ee2b89187" Jul 21 02:10:00.874017 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:00.873933 2574 generic.go:356] "Generic (PLEG): container finished" podID="3504ec09-d4b6-41ce-b77d-d6c73b437a33" containerID="1947582b743b6434949d3c938542cbc3cb18979368ee3876f486d8c7fae57990" exitCode=0 Jul 21 02:10:00.874017 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:00.873993 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x2w8b" event={"ID":"3504ec09-d4b6-41ce-b77d-d6c73b437a33","Type":"ContainerDied","Data":"1947582b743b6434949d3c938542cbc3cb18979368ee3876f486d8c7fae57990"} Jul 21 02:10:01.043890 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:01.043862 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03e524b0-40bd-46b4-ac8f-eb346e94a7dc-metrics-certs\") pod \"network-metrics-daemon-8v67q\" (UID: \"03e524b0-40bd-46b4-ac8f-eb346e94a7dc\") " pod="openshift-multus/network-metrics-daemon-8v67q" Jul 21 02:10:01.044052 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:10:01.043972 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 02:10:01.044052 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:10:01.044036 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03e524b0-40bd-46b4-ac8f-eb346e94a7dc-metrics-certs podName:03e524b0-40bd-46b4-ac8f-eb346e94a7dc nodeName:}" failed. No retries permitted until 2026-07-21 02:10:33.044019075 +0000 UTC m=+67.063512974 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03e524b0-40bd-46b4-ac8f-eb346e94a7dc-metrics-certs") pod "network-metrics-daemon-8v67q" (UID: "03e524b0-40bd-46b4-ac8f-eb346e94a7dc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 02:10:01.144407 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:01.144319 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4bee3776-d7db-4be5-8e76-337ee2b89187-original-pull-secret\") pod \"global-pull-secret-syncer-ljdcc\" (UID: \"4bee3776-d7db-4be5-8e76-337ee2b89187\") " pod="kube-system/global-pull-secret-syncer-ljdcc" Jul 21 02:10:01.144538 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:10:01.144438 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 02:10:01.144538 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:10:01.144492 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bee3776-d7db-4be5-8e76-337ee2b89187-original-pull-secret podName:4bee3776-d7db-4be5-8e76-337ee2b89187 nodeName:}" failed. No retries permitted until 2026-07-21 02:10:33.144479612 +0000 UTC m=+67.163973494 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4bee3776-d7db-4be5-8e76-337ee2b89187-original-pull-secret") pod "global-pull-secret-syncer-ljdcc" (UID: "4bee3776-d7db-4be5-8e76-337ee2b89187") : object "kube-system"/"original-pull-secret" not registered Jul 21 02:10:01.245119 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:01.245087 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht26w\" (UniqueName: \"kubernetes.io/projected/80deee90-4b47-4cdf-bc87-f9242ad63cec-kube-api-access-ht26w\") pod \"network-check-target-qqnx4\" (UID: \"80deee90-4b47-4cdf-bc87-f9242ad63cec\") " pod="openshift-network-diagnostics/network-check-target-qqnx4" Jul 21 02:10:01.245319 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:10:01.245298 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 02:10:01.245383 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:10:01.245324 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 02:10:01.245383 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:10:01.245336 2574 projected.go:196] Error preparing data for projected volume kube-api-access-ht26w for pod openshift-network-diagnostics/network-check-target-qqnx4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 02:10:01.245445 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:10:01.245390 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/80deee90-4b47-4cdf-bc87-f9242ad63cec-kube-api-access-ht26w podName:80deee90-4b47-4cdf-bc87-f9242ad63cec nodeName:}" failed. No retries permitted until 2026-07-21 02:10:33.245376503 +0000 UTC m=+67.264870399 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-ht26w" (UniqueName: "kubernetes.io/projected/80deee90-4b47-4cdf-bc87-f9242ad63cec-kube-api-access-ht26w") pod "network-check-target-qqnx4" (UID: "80deee90-4b47-4cdf-bc87-f9242ad63cec") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 02:10:01.479174 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:01.479091 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqnx4" Jul 21 02:10:01.479563 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:10:01.479229 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qqnx4" podUID="80deee90-4b47-4cdf-bc87-f9242ad63cec" Jul 21 02:10:02.479173 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:02.479110 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ljdcc" Jul 21 02:10:02.479611 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:10:02.479258 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ljdcc" podUID="4bee3776-d7db-4be5-8e76-337ee2b89187" Jul 21 02:10:02.479611 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:02.479292 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8v67q" Jul 21 02:10:02.479611 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:10:02.479417 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8v67q" podUID="03e524b0-40bd-46b4-ac8f-eb346e94a7dc" Jul 21 02:10:03.240123 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:03.240040 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-37.ec2.internal" event="NodeReady" Jul 21 02:10:03.240295 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:03.240211 2574 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 21 02:10:03.277574 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:03.277542 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7847d77f6d-9wndp"] Jul 21 02:10:03.279224 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:03.279204 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7847d77f6d-9wndp" Jul 21 02:10:03.285415 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:03.285387 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 21 02:10:03.285415 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:03.285404 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 21 02:10:03.285637 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:03.285420 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6qdf5\"" Jul 21 02:10:03.286299 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:03.286256 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 21 02:10:03.294085 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:03.294062 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 21 02:10:03.301163 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:03.301137 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-24k47"] Jul 21 02:10:03.303338 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:03.303309 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-pc4fb"] Jul 21 02:10:03.303489 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:03.303460 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-24k47" Jul 21 02:10:03.305150 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:03.305132 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7847d77f6d-9wndp"] Jul 21 02:10:03.305257 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:03.305227 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pc4fb" Jul 21 02:10:03.306173 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:03.306154 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 21 02:10:03.306291 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:03.306232 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 21 02:10:03.306562 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:03.306546 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lx8r6\"" Jul 21 02:10:03.307622 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:03.307593 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 21 02:10:03.307895 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:03.307878 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 21 02:10:03.307962 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:03.307911 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-9wsng\"" Jul 21 02:10:03.308482 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:03.308465 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 21 02:10:03.317897 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:03.317855 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-24k47"] Jul 21 02:10:03.318809 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:03.318789 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pc4fb"] Jul 21 02:10:03.360205 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:03.360171 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5afc6d56-b4d0-46f3-958d-ea3b873ee718-registry-certificates\") pod \"image-registry-7847d77f6d-9wndp\" (UID: \"5afc6d56-b4d0-46f3-958d-ea3b873ee718\") " pod="openshift-image-registry/image-registry-7847d77f6d-9wndp" Jul 21 02:10:03.360385 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:03.360215 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5afc6d56-b4d0-46f3-958d-ea3b873ee718-bound-sa-token\") pod \"image-registry-7847d77f6d-9wndp\" (UID: \"5afc6d56-b4d0-46f3-958d-ea3b873ee718\") " pod="openshift-image-registry/image-registry-7847d77f6d-9wndp" Jul 21 02:10:03.360385 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:03.360280 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5afc6d56-b4d0-46f3-958d-ea3b873ee718-ca-trust-extracted\") pod \"image-registry-7847d77f6d-9wndp\" (UID: \"5afc6d56-b4d0-46f3-958d-ea3b873ee718\") " pod="openshift-image-registry/image-registry-7847d77f6d-9wndp" Jul 21 02:10:03.360385 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:03.360348 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5afc6d56-b4d0-46f3-958d-ea3b873ee718-image-registry-private-configuration\") pod \"image-registry-7847d77f6d-9wndp\" (UID: \"5afc6d56-b4d0-46f3-958d-ea3b873ee718\") " pod="openshift-image-registry/image-registry-7847d77f6d-9wndp" Jul 21 02:10:03.360527 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:03.360388 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5afc6d56-b4d0-46f3-958d-ea3b873ee718-registry-tls\") pod \"image-registry-7847d77f6d-9wndp\" (UID: \"5afc6d56-b4d0-46f3-958d-ea3b873ee718\") " pod="openshift-image-registry/image-registry-7847d77f6d-9wndp" Jul 21 02:10:03.360527 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:03.360415 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5afc6d56-b4d0-46f3-958d-ea3b873ee718-installation-pull-secrets\") pod \"image-registry-7847d77f6d-9wndp\" (UID: \"5afc6d56-b4d0-46f3-958d-ea3b873ee718\") " pod="openshift-image-registry/image-registry-7847d77f6d-9wndp" Jul 21 02:10:03.360527 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:03.360461 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5afc6d56-b4d0-46f3-958d-ea3b873ee718-trusted-ca\") pod \"image-registry-7847d77f6d-9wndp\" (UID: \"5afc6d56-b4d0-46f3-958d-ea3b873ee718\") " pod="openshift-image-registry/image-registry-7847d77f6d-9wndp" Jul 21 02:10:03.360527 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:03.360503 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kng9\" (UniqueName: \"kubernetes.io/projected/5afc6d56-b4d0-46f3-958d-ea3b873ee718-kube-api-access-2kng9\") pod \"image-registry-7847d77f6d-9wndp\" (UID: \"5afc6d56-b4d0-46f3-958d-ea3b873ee718\") " pod="openshift-image-registry/image-registry-7847d77f6d-9wndp" Jul 21 02:10:03.461406 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:03.461375 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5afc6d56-b4d0-46f3-958d-ea3b873ee718-ca-trust-extracted\") pod \"image-registry-7847d77f6d-9wndp\" (UID: \"5afc6d56-b4d0-46f3-958d-ea3b873ee718\") " pod="openshift-image-registry/image-registry-7847d77f6d-9wndp" Jul 21 02:10:03.461575 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:03.461422 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5afc6d56-b4d0-46f3-958d-ea3b873ee718-image-registry-private-configuration\") pod \"image-registry-7847d77f6d-9wndp\" (UID: \"5afc6d56-b4d0-46f3-958d-ea3b873ee718\") " pod="openshift-image-registry/image-registry-7847d77f6d-9wndp" Jul 21 02:10:03.461575 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:03.461538 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5afc6d56-b4d0-46f3-958d-ea3b873ee718-registry-tls\") pod \"image-registry-7847d77f6d-9wndp\" (UID: \"5afc6d56-b4d0-46f3-958d-ea3b873ee718\") " pod="openshift-image-registry/image-registry-7847d77f6d-9wndp" Jul 21 02:10:03.461575 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:03.461570 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5afc6d56-b4d0-46f3-958d-ea3b873ee718-installation-pull-secrets\") pod \"image-registry-7847d77f6d-9wndp\" (UID: \"5afc6d56-b4d0-46f3-958d-ea3b873ee718\") " pod="openshift-image-registry/image-registry-7847d77f6d-9wndp" Jul 21 02:10:03.461727 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:03.461597 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5afc6d56-b4d0-46f3-958d-ea3b873ee718-trusted-ca\") pod \"image-registry-7847d77f6d-9wndp\" (UID: \"5afc6d56-b4d0-46f3-958d-ea3b873ee718\") " pod="openshift-image-registry/image-registry-7847d77f6d-9wndp" Jul 21 02:10:03.461727 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:03.461622 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f7a19bce-85cf-4f26-aafa-977786cbfc4d-metrics-tls\") pod \"dns-default-24k47\" (UID: \"f7a19bce-85cf-4f26-aafa-977786cbfc4d\") " pod="openshift-dns/dns-default-24k47" Jul 21 02:10:03.461727 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:10:03.461702 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 02:10:03.461878 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:10:03.461729 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7847d77f6d-9wndp: secret "image-registry-tls" not found Jul 21 02:10:03.461878 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:03.461736 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7a19bce-85cf-4f26-aafa-977786cbfc4d-tmp-dir\") pod \"dns-default-24k47\" (UID: \"f7a19bce-85cf-4f26-aafa-977786cbfc4d\") " pod="openshift-dns/dns-default-24k47" Jul 21 02:10:03.461878 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:03.461781 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kng9\" (UniqueName: \"kubernetes.io/projected/5afc6d56-b4d0-46f3-958d-ea3b873ee718-kube-api-access-2kng9\") pod \"image-registry-7847d77f6d-9wndp\" (UID: \"5afc6d56-b4d0-46f3-958d-ea3b873ee718\") " pod="openshift-image-registry/image-registry-7847d77f6d-9wndp" Jul 21 02:10:03.461878 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:10:03.461799 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5afc6d56-b4d0-46f3-958d-ea3b873ee718-registry-tls podName:5afc6d56-b4d0-46f3-958d-ea3b873ee718 nodeName:}" failed. No retries permitted until 2026-07-21 02:10:03.961777002 +0000 UTC m=+37.981270887 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5afc6d56-b4d0-46f3-958d-ea3b873ee718-registry-tls") pod "image-registry-7847d77f6d-9wndp" (UID: "5afc6d56-b4d0-46f3-958d-ea3b873ee718") : secret "image-registry-tls" not found Jul 21 02:10:03.461878 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:03.461832 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f7a19bce-85cf-4f26-aafa-977786cbfc4d-config-volume\") pod \"dns-default-24k47\" (UID: \"f7a19bce-85cf-4f26-aafa-977786cbfc4d\") " pod="openshift-dns/dns-default-24k47" Jul 21 02:10:03.462128 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:03.461905 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/224189a4-715f-4007-8d45-832188c1f1d1-cert\") pod \"ingress-canary-pc4fb\" (UID: \"224189a4-715f-4007-8d45-832188c1f1d1\") " pod="openshift-ingress-canary/ingress-canary-pc4fb" Jul 21 02:10:03.462128 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:03.461922 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5afc6d56-b4d0-46f3-958d-ea3b873ee718-ca-trust-extracted\") pod \"image-registry-7847d77f6d-9wndp\" (UID: \"5afc6d56-b4d0-46f3-958d-ea3b873ee718\") " pod="openshift-image-registry/image-registry-7847d77f6d-9wndp" Jul 21 02:10:03.462128 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:03.461937 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxprz\" (UniqueName: \"kubernetes.io/projected/f7a19bce-85cf-4f26-aafa-977786cbfc4d-kube-api-access-rxprz\") pod \"dns-default-24k47\" (UID: \"f7a19bce-85cf-4f26-aafa-977786cbfc4d\") " pod="openshift-dns/dns-default-24k47" Jul 21 02:10:03.462128 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:03.461996 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5afc6d56-b4d0-46f3-958d-ea3b873ee718-registry-certificates\") pod \"image-registry-7847d77f6d-9wndp\" (UID: \"5afc6d56-b4d0-46f3-958d-ea3b873ee718\") " pod="openshift-image-registry/image-registry-7847d77f6d-9wndp" Jul 21 02:10:03.462128 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:03.462022 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5afc6d56-b4d0-46f3-958d-ea3b873ee718-bound-sa-token\") pod \"image-registry-7847d77f6d-9wndp\" (UID: \"5afc6d56-b4d0-46f3-958d-ea3b873ee718\") " pod="openshift-image-registry/image-registry-7847d77f6d-9wndp" Jul 21 02:10:03.462409 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:03.462198 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klpg4\" (UniqueName: \"kubernetes.io/projected/224189a4-715f-4007-8d45-832188c1f1d1-kube-api-access-klpg4\") pod \"ingress-canary-pc4fb\" (UID: \"224189a4-715f-4007-8d45-832188c1f1d1\") " pod="openshift-ingress-canary/ingress-canary-pc4fb" Jul 21 02:10:03.462537 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:03.462517 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5afc6d56-b4d0-46f3-958d-ea3b873ee718-registry-certificates\") pod \"image-registry-7847d77f6d-9wndp\" (UID: \"5afc6d56-b4d0-46f3-958d-ea3b873ee718\") " pod="openshift-image-registry/image-registry-7847d77f6d-9wndp" Jul 21 02:10:03.462819 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:03.462788 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5afc6d56-b4d0-46f3-958d-ea3b873ee718-trusted-ca\") pod \"image-registry-7847d77f6d-9wndp\" (UID: \"5afc6d56-b4d0-46f3-958d-ea3b873ee718\") " pod="openshift-image-registry/image-registry-7847d77f6d-9wndp" Jul 21 02:10:03.465872 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:03.465768 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5afc6d56-b4d0-46f3-958d-ea3b873ee718-image-registry-private-configuration\") pod \"image-registry-7847d77f6d-9wndp\" (UID: \"5afc6d56-b4d0-46f3-958d-ea3b873ee718\") " pod="openshift-image-registry/image-registry-7847d77f6d-9wndp" Jul 21 02:10:03.465963 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:03.465770 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5afc6d56-b4d0-46f3-958d-ea3b873ee718-installation-pull-secrets\") pod \"image-registry-7847d77f6d-9wndp\" (UID: \"5afc6d56-b4d0-46f3-958d-ea3b873ee718\") " pod="openshift-image-registry/image-registry-7847d77f6d-9wndp" Jul 21 02:10:03.471397 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:03.471374 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5afc6d56-b4d0-46f3-958d-ea3b873ee718-bound-sa-token\") pod \"image-registry-7847d77f6d-9wndp\" (UID: \"5afc6d56-b4d0-46f3-958d-ea3b873ee718\") " pod="openshift-image-registry/image-registry-7847d77f6d-9wndp" Jul 21 02:10:03.471863 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:03.471841 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kng9\" (UniqueName: \"kubernetes.io/projected/5afc6d56-b4d0-46f3-958d-ea3b873ee718-kube-api-access-2kng9\") pod \"image-registry-7847d77f6d-9wndp\" (UID: \"5afc6d56-b4d0-46f3-958d-ea3b873ee718\") " pod="openshift-image-registry/image-registry-7847d77f6d-9wndp" Jul 21 02:10:03.479355 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:03.479335 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqnx4" Jul 21 02:10:03.482076 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:03.482061 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 21 02:10:03.482180 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:03.482150 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gcvct\"" Jul 21 02:10:03.482425 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:03.482398 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 21 02:10:03.563112 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:03.563017 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f7a19bce-85cf-4f26-aafa-977786cbfc4d-metrics-tls\") pod \"dns-default-24k47\" (UID: \"f7a19bce-85cf-4f26-aafa-977786cbfc4d\") " pod="openshift-dns/dns-default-24k47" Jul 21 02:10:03.563311 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:10:03.563149 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 02:10:03.563311 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:03.563190 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7a19bce-85cf-4f26-aafa-977786cbfc4d-tmp-dir\") pod \"dns-default-24k47\" (UID: \"f7a19bce-85cf-4f26-aafa-977786cbfc4d\") " pod="openshift-dns/dns-default-24k47" Jul 21 02:10:03.563311 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:10:03.563227 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7a19bce-85cf-4f26-aafa-977786cbfc4d-metrics-tls podName:f7a19bce-85cf-4f26-aafa-977786cbfc4d nodeName:}" failed. No retries permitted until 2026-07-21 02:10:04.06320623 +0000 UTC m=+38.082700116 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f7a19bce-85cf-4f26-aafa-977786cbfc4d-metrics-tls") pod "dns-default-24k47" (UID: "f7a19bce-85cf-4f26-aafa-977786cbfc4d") : secret "dns-default-metrics-tls" not found Jul 21 02:10:03.563311 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:03.563295 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f7a19bce-85cf-4f26-aafa-977786cbfc4d-config-volume\") pod \"dns-default-24k47\" (UID: \"f7a19bce-85cf-4f26-aafa-977786cbfc4d\") " pod="openshift-dns/dns-default-24k47" Jul 21 02:10:03.563535 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:03.563362 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/224189a4-715f-4007-8d45-832188c1f1d1-cert\") pod \"ingress-canary-pc4fb\" (UID: \"224189a4-715f-4007-8d45-832188c1f1d1\") " pod="openshift-ingress-canary/ingress-canary-pc4fb" Jul 21 02:10:03.563535 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:03.563392 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxprz\" (UniqueName: \"kubernetes.io/projected/f7a19bce-85cf-4f26-aafa-977786cbfc4d-kube-api-access-rxprz\") pod \"dns-default-24k47\" (UID: \"f7a19bce-85cf-4f26-aafa-977786cbfc4d\") " pod="openshift-dns/dns-default-24k47" Jul 21 02:10:03.563535 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:03.563427 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klpg4\" (UniqueName: \"kubernetes.io/projected/224189a4-715f-4007-8d45-832188c1f1d1-kube-api-access-klpg4\") pod \"ingress-canary-pc4fb\" (UID: \"224189a4-715f-4007-8d45-832188c1f1d1\") " pod="openshift-ingress-canary/ingress-canary-pc4fb" Jul 21 02:10:03.563677 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:03.563559 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7a19bce-85cf-4f26-aafa-977786cbfc4d-tmp-dir\") pod \"dns-default-24k47\" (UID: \"f7a19bce-85cf-4f26-aafa-977786cbfc4d\") " pod="openshift-dns/dns-default-24k47" Jul 21 02:10:03.563925 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:03.563902 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f7a19bce-85cf-4f26-aafa-977786cbfc4d-config-volume\") pod \"dns-default-24k47\" (UID: \"f7a19bce-85cf-4f26-aafa-977786cbfc4d\") " pod="openshift-dns/dns-default-24k47" Jul 21 02:10:03.566202 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:03.566179 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/224189a4-715f-4007-8d45-832188c1f1d1-cert\") pod \"ingress-canary-pc4fb\" (UID: \"224189a4-715f-4007-8d45-832188c1f1d1\") " pod="openshift-ingress-canary/ingress-canary-pc4fb" Jul 21 02:10:03.572633 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:03.572608 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klpg4\" (UniqueName: \"kubernetes.io/projected/224189a4-715f-4007-8d45-832188c1f1d1-kube-api-access-klpg4\") pod \"ingress-canary-pc4fb\" (UID: \"224189a4-715f-4007-8d45-832188c1f1d1\") " pod="openshift-ingress-canary/ingress-canary-pc4fb" Jul 21 02:10:03.575050 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:03.575029 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxprz\" (UniqueName: \"kubernetes.io/projected/f7a19bce-85cf-4f26-aafa-977786cbfc4d-kube-api-access-rxprz\") pod \"dns-default-24k47\" (UID: \"f7a19bce-85cf-4f26-aafa-977786cbfc4d\") " pod="openshift-dns/dns-default-24k47" Jul 21 02:10:03.622339 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:03.622301 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pc4fb" Jul 21 02:10:03.966114 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:03.966019 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5afc6d56-b4d0-46f3-958d-ea3b873ee718-registry-tls\") pod \"image-registry-7847d77f6d-9wndp\" (UID: \"5afc6d56-b4d0-46f3-958d-ea3b873ee718\") " pod="openshift-image-registry/image-registry-7847d77f6d-9wndp" Jul 21 02:10:03.966293 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:10:03.966207 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 02:10:03.966293 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:10:03.966229 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7847d77f6d-9wndp: secret "image-registry-tls" not found Jul 21 02:10:03.966412 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:10:03.966317 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5afc6d56-b4d0-46f3-958d-ea3b873ee718-registry-tls podName:5afc6d56-b4d0-46f3-958d-ea3b873ee718 nodeName:}" failed. No retries permitted until 2026-07-21 02:10:04.966292165 +0000 UTC m=+38.985786068 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5afc6d56-b4d0-46f3-958d-ea3b873ee718-registry-tls") pod "image-registry-7847d77f6d-9wndp" (UID: "5afc6d56-b4d0-46f3-958d-ea3b873ee718") : secret "image-registry-tls" not found Jul 21 02:10:04.067470 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:04.067428 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f7a19bce-85cf-4f26-aafa-977786cbfc4d-metrics-tls\") pod \"dns-default-24k47\" (UID: \"f7a19bce-85cf-4f26-aafa-977786cbfc4d\") " pod="openshift-dns/dns-default-24k47" Jul 21 02:10:04.067636 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:10:04.067617 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 02:10:04.067714 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:10:04.067701 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7a19bce-85cf-4f26-aafa-977786cbfc4d-metrics-tls podName:f7a19bce-85cf-4f26-aafa-977786cbfc4d nodeName:}" failed. No retries permitted until 2026-07-21 02:10:05.067679565 +0000 UTC m=+39.087173454 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f7a19bce-85cf-4f26-aafa-977786cbfc4d-metrics-tls") pod "dns-default-24k47" (UID: "f7a19bce-85cf-4f26-aafa-977786cbfc4d") : secret "dns-default-metrics-tls" not found Jul 21 02:10:04.478817 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:04.478777 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8v67q" Jul 21 02:10:04.479009 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:04.478989 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ljdcc" Jul 21 02:10:04.481697 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:04.481667 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 21 02:10:04.482078 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:04.481745 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 21 02:10:04.482078 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:04.481798 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2zscq\"" Jul 21 02:10:04.975368 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:04.975338 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5afc6d56-b4d0-46f3-958d-ea3b873ee718-registry-tls\") pod \"image-registry-7847d77f6d-9wndp\" (UID: \"5afc6d56-b4d0-46f3-958d-ea3b873ee718\") " pod="openshift-image-registry/image-registry-7847d77f6d-9wndp" Jul 21 02:10:04.975595 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:10:04.975566 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 02:10:04.975595 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:10:04.975582 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7847d77f6d-9wndp: secret "image-registry-tls" not found Jul 21 02:10:04.975928 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:10:04.975908 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5afc6d56-b4d0-46f3-958d-ea3b873ee718-registry-tls podName:5afc6d56-b4d0-46f3-958d-ea3b873ee718 nodeName:}" failed. No retries permitted until 2026-07-21 02:10:06.975885337 +0000 UTC m=+40.995379236 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5afc6d56-b4d0-46f3-958d-ea3b873ee718-registry-tls") pod "image-registry-7847d77f6d-9wndp" (UID: "5afc6d56-b4d0-46f3-958d-ea3b873ee718") : secret "image-registry-tls" not found Jul 21 02:10:05.022006 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:05.021973 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pc4fb"] Jul 21 02:10:05.025646 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:10:05.025601 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod224189a4_715f_4007_8d45_832188c1f1d1.slice/crio-927978e8d24307d76a5c1264d6225ce17793ded3dd65b246da95e26ef34125b3 WatchSource:0}: Error finding container 927978e8d24307d76a5c1264d6225ce17793ded3dd65b246da95e26ef34125b3: Status 404 returned error can't find the container with id 927978e8d24307d76a5c1264d6225ce17793ded3dd65b246da95e26ef34125b3 Jul 21 02:10:05.076617 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:05.076580 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f7a19bce-85cf-4f26-aafa-977786cbfc4d-metrics-tls\") pod \"dns-default-24k47\" (UID: \"f7a19bce-85cf-4f26-aafa-977786cbfc4d\") " pod="openshift-dns/dns-default-24k47" Jul 21 02:10:05.076771 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:10:05.076745 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 02:10:05.076816 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:10:05.076805 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7a19bce-85cf-4f26-aafa-977786cbfc4d-metrics-tls podName:f7a19bce-85cf-4f26-aafa-977786cbfc4d nodeName:}" failed. No retries permitted until 2026-07-21 02:10:07.076791457 +0000 UTC m=+41.096285344 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f7a19bce-85cf-4f26-aafa-977786cbfc4d-metrics-tls") pod "dns-default-24k47" (UID: "f7a19bce-85cf-4f26-aafa-977786cbfc4d") : secret "dns-default-metrics-tls" not found Jul 21 02:10:05.890042 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:05.890011 2574 generic.go:356] "Generic (PLEG): container finished" podID="3504ec09-d4b6-41ce-b77d-d6c73b437a33" containerID="d170cd64e9d595ad9b8589568a4185948e8c512445ca885d9623a3895c0a9795" exitCode=0 Jul 21 02:10:05.890847 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:05.890089 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x2w8b" event={"ID":"3504ec09-d4b6-41ce-b77d-d6c73b437a33","Type":"ContainerDied","Data":"d170cd64e9d595ad9b8589568a4185948e8c512445ca885d9623a3895c0a9795"} Jul 21 02:10:05.891964 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:05.891744 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pc4fb" event={"ID":"224189a4-715f-4007-8d45-832188c1f1d1","Type":"ContainerStarted","Data":"927978e8d24307d76a5c1264d6225ce17793ded3dd65b246da95e26ef34125b3"} Jul 21 02:10:06.898567 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:06.898533 2574 generic.go:356] "Generic (PLEG): container finished" podID="3504ec09-d4b6-41ce-b77d-d6c73b437a33" containerID="a14b8acbbbffb4892cc9fb67391e10871874e8eb33d1976bcfcc4b7760f695c3" exitCode=0 Jul 21 02:10:06.898995 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:06.898607 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x2w8b" event={"ID":"3504ec09-d4b6-41ce-b77d-d6c73b437a33","Type":"ContainerDied","Data":"a14b8acbbbffb4892cc9fb67391e10871874e8eb33d1976bcfcc4b7760f695c3"} Jul 21 02:10:06.993884 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:06.993845 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5afc6d56-b4d0-46f3-958d-ea3b873ee718-registry-tls\") pod \"image-registry-7847d77f6d-9wndp\" (UID: \"5afc6d56-b4d0-46f3-958d-ea3b873ee718\") " pod="openshift-image-registry/image-registry-7847d77f6d-9wndp" Jul 21 02:10:06.994062 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:10:06.993956 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 02:10:06.994062 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:10:06.993977 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7847d77f6d-9wndp: secret "image-registry-tls" not found Jul 21 02:10:06.994062 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:10:06.994032 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5afc6d56-b4d0-46f3-958d-ea3b873ee718-registry-tls podName:5afc6d56-b4d0-46f3-958d-ea3b873ee718 nodeName:}" failed. No retries permitted until 2026-07-21 02:10:10.994011942 +0000 UTC m=+45.013505825 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5afc6d56-b4d0-46f3-958d-ea3b873ee718-registry-tls") pod "image-registry-7847d77f6d-9wndp" (UID: "5afc6d56-b4d0-46f3-958d-ea3b873ee718") : secret "image-registry-tls" not found Jul 21 02:10:07.095080 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:07.095039 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f7a19bce-85cf-4f26-aafa-977786cbfc4d-metrics-tls\") pod \"dns-default-24k47\" (UID: \"f7a19bce-85cf-4f26-aafa-977786cbfc4d\") " pod="openshift-dns/dns-default-24k47" Jul 21 02:10:07.095215 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:10:07.095192 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 02:10:07.095298 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:10:07.095280 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7a19bce-85cf-4f26-aafa-977786cbfc4d-metrics-tls podName:f7a19bce-85cf-4f26-aafa-977786cbfc4d nodeName:}" failed. No retries permitted until 2026-07-21 02:10:11.095246479 +0000 UTC m=+45.114740361 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f7a19bce-85cf-4f26-aafa-977786cbfc4d-metrics-tls") pod "dns-default-24k47" (UID: "f7a19bce-85cf-4f26-aafa-977786cbfc4d") : secret "dns-default-metrics-tls" not found Jul 21 02:10:07.903742 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:07.903709 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x2w8b" event={"ID":"3504ec09-d4b6-41ce-b77d-d6c73b437a33","Type":"ContainerStarted","Data":"04701baf6a848cefc19753710351644f2ec332ef9d51251f32ce427dbbc589a3"} Jul 21 02:10:07.905081 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:07.905059 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pc4fb" event={"ID":"224189a4-715f-4007-8d45-832188c1f1d1","Type":"ContainerStarted","Data":"e1d2a754bd7e9c3a7fb068fb563a96adb836064820a213a8b7c37cc7d1c295ca"} Jul 21 02:10:07.953171 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:07.953122 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-x2w8b" podStartSLOduration=9.214705137 podStartE2EDuration="39.95310614s" totalImagesPullingTime="30.738401003s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 02:09:28 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:10:07.95245507 +0000 UTC m=+41.971948976" watchObservedRunningTime="2026-07-21 02:10:07.95310614 +0000 UTC m=+41.972600046" Jul 21 02:10:07.982642 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:07.982593 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-pc4fb" podStartSLOduration=2.698589845 podStartE2EDuration="4.982580054s" totalImagesPullingTime="2.283990209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 02:10:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:10:07.982369478 +0000 UTC m=+42.001863383" watchObservedRunningTime="2026-07-21 02:10:07.982580054 +0000 UTC m=+42.002073978" Jul 21 02:10:11.021144 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:11.021103 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5afc6d56-b4d0-46f3-958d-ea3b873ee718-registry-tls\") pod \"image-registry-7847d77f6d-9wndp\" (UID: \"5afc6d56-b4d0-46f3-958d-ea3b873ee718\") " pod="openshift-image-registry/image-registry-7847d77f6d-9wndp" Jul 21 02:10:11.021546 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:10:11.021281 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 02:10:11.021546 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:10:11.021302 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7847d77f6d-9wndp: secret "image-registry-tls" not found Jul 21 02:10:11.021546 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:10:11.021372 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5afc6d56-b4d0-46f3-958d-ea3b873ee718-registry-tls podName:5afc6d56-b4d0-46f3-958d-ea3b873ee718 nodeName:}" failed. No retries permitted until 2026-07-21 02:10:19.021356364 +0000 UTC m=+53.040850250 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5afc6d56-b4d0-46f3-958d-ea3b873ee718-registry-tls") pod "image-registry-7847d77f6d-9wndp" (UID: "5afc6d56-b4d0-46f3-958d-ea3b873ee718") : secret "image-registry-tls" not found Jul 21 02:10:11.122081 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:11.122047 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f7a19bce-85cf-4f26-aafa-977786cbfc4d-metrics-tls\") pod \"dns-default-24k47\" (UID: \"f7a19bce-85cf-4f26-aafa-977786cbfc4d\") " pod="openshift-dns/dns-default-24k47" Jul 21 02:10:11.122221 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:10:11.122197 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 02:10:11.122290 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:10:11.122278 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7a19bce-85cf-4f26-aafa-977786cbfc4d-metrics-tls podName:f7a19bce-85cf-4f26-aafa-977786cbfc4d nodeName:}" failed. No retries permitted until 2026-07-21 02:10:19.122248557 +0000 UTC m=+53.141742439 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f7a19bce-85cf-4f26-aafa-977786cbfc4d-metrics-tls") pod "dns-default-24k47" (UID: "f7a19bce-85cf-4f26-aafa-977786cbfc4d") : secret "dns-default-metrics-tls" not found Jul 21 02:10:11.791576 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:11.791542 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-ncxvc"] Jul 21 02:10:11.823378 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:11.823342 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ncxvc"] Jul 21 02:10:11.823522 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:11.823466 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ncxvc" Jul 21 02:10:11.827740 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:11.827717 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 21 02:10:11.828277 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:11.828249 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 21 02:10:11.830705 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:11.830684 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wj6qb\"" Jul 21 02:10:11.831400 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:11.831383 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 21 02:10:11.831764 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:11.831748 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 21 02:10:11.928619 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:11.928591 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a89ab6c0-0bd2-4cb5-aca9-71b07c462005-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ncxvc\" (UID: \"a89ab6c0-0bd2-4cb5-aca9-71b07c462005\") " pod="openshift-insights/insights-runtime-extractor-ncxvc" Jul 21 02:10:11.928787 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:11.928626 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a89ab6c0-0bd2-4cb5-aca9-71b07c462005-crio-socket\") pod \"insights-runtime-extractor-ncxvc\" (UID: \"a89ab6c0-0bd2-4cb5-aca9-71b07c462005\") " pod="openshift-insights/insights-runtime-extractor-ncxvc" Jul 21 02:10:11.928787 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:11.928646 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a89ab6c0-0bd2-4cb5-aca9-71b07c462005-data-volume\") pod \"insights-runtime-extractor-ncxvc\" (UID: \"a89ab6c0-0bd2-4cb5-aca9-71b07c462005\") " pod="openshift-insights/insights-runtime-extractor-ncxvc" Jul 21 02:10:11.928787 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:11.928723 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a89ab6c0-0bd2-4cb5-aca9-71b07c462005-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ncxvc\" (UID: \"a89ab6c0-0bd2-4cb5-aca9-71b07c462005\") " pod="openshift-insights/insights-runtime-extractor-ncxvc" Jul 21 02:10:11.928787 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:11.928779 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ccvk\" (UniqueName: \"kubernetes.io/projected/a89ab6c0-0bd2-4cb5-aca9-71b07c462005-kube-api-access-6ccvk\") pod \"insights-runtime-extractor-ncxvc\" (UID: \"a89ab6c0-0bd2-4cb5-aca9-71b07c462005\") " pod="openshift-insights/insights-runtime-extractor-ncxvc" Jul 21 02:10:12.029358 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:12.029320 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a89ab6c0-0bd2-4cb5-aca9-71b07c462005-crio-socket\") pod \"insights-runtime-extractor-ncxvc\" (UID: \"a89ab6c0-0bd2-4cb5-aca9-71b07c462005\") " pod="openshift-insights/insights-runtime-extractor-ncxvc" Jul 21 02:10:12.029358 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:12.029361 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a89ab6c0-0bd2-4cb5-aca9-71b07c462005-data-volume\") pod \"insights-runtime-extractor-ncxvc\" (UID: \"a89ab6c0-0bd2-4cb5-aca9-71b07c462005\") " pod="openshift-insights/insights-runtime-extractor-ncxvc" Jul 21 02:10:12.029797 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:12.029385 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a89ab6c0-0bd2-4cb5-aca9-71b07c462005-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ncxvc\" (UID: \"a89ab6c0-0bd2-4cb5-aca9-71b07c462005\") " pod="openshift-insights/insights-runtime-extractor-ncxvc" Jul 21 02:10:12.029797 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:12.029412 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ccvk\" (UniqueName: \"kubernetes.io/projected/a89ab6c0-0bd2-4cb5-aca9-71b07c462005-kube-api-access-6ccvk\") pod \"insights-runtime-extractor-ncxvc\" (UID: \"a89ab6c0-0bd2-4cb5-aca9-71b07c462005\") " pod="openshift-insights/insights-runtime-extractor-ncxvc" Jul 21 02:10:12.029797 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:12.029493 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a89ab6c0-0bd2-4cb5-aca9-71b07c462005-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ncxvc\" (UID: \"a89ab6c0-0bd2-4cb5-aca9-71b07c462005\") " pod="openshift-insights/insights-runtime-extractor-ncxvc" Jul 21 02:10:12.029797 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:12.029524 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a89ab6c0-0bd2-4cb5-aca9-71b07c462005-crio-socket\") pod \"insights-runtime-extractor-ncxvc\" (UID: \"a89ab6c0-0bd2-4cb5-aca9-71b07c462005\") " pod="openshift-insights/insights-runtime-extractor-ncxvc" Jul 21 02:10:12.029797 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:10:12.029564 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 21 02:10:12.029797 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:10:12.029631 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a89ab6c0-0bd2-4cb5-aca9-71b07c462005-kube-rbac-proxy-cm podName:a89ab6c0-0bd2-4cb5-aca9-71b07c462005 nodeName:}" failed. No retries permitted until 2026-07-21 02:10:12.529614193 +0000 UTC m=+46.549108076 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a89ab6c0-0bd2-4cb5-aca9-71b07c462005-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ncxvc" (UID: "a89ab6c0-0bd2-4cb5-aca9-71b07c462005") : configmap "kube-rbac-proxy" not found Jul 21 02:10:12.029797 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:10:12.029565 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 02:10:12.029797 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:10:12.029724 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a89ab6c0-0bd2-4cb5-aca9-71b07c462005-insights-runtime-extractor-tls podName:a89ab6c0-0bd2-4cb5-aca9-71b07c462005 nodeName:}" failed. No retries permitted until 2026-07-21 02:10:12.529703242 +0000 UTC m=+46.549197142 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a89ab6c0-0bd2-4cb5-aca9-71b07c462005-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ncxvc" (UID: "a89ab6c0-0bd2-4cb5-aca9-71b07c462005") : secret "insights-runtime-extractor-tls" not found Jul 21 02:10:12.029797 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:12.029753 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a89ab6c0-0bd2-4cb5-aca9-71b07c462005-data-volume\") pod \"insights-runtime-extractor-ncxvc\" (UID: \"a89ab6c0-0bd2-4cb5-aca9-71b07c462005\") " pod="openshift-insights/insights-runtime-extractor-ncxvc" Jul 21 02:10:12.041419 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:12.041391 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ccvk\" (UniqueName: \"kubernetes.io/projected/a89ab6c0-0bd2-4cb5-aca9-71b07c462005-kube-api-access-6ccvk\") pod \"insights-runtime-extractor-ncxvc\" (UID: \"a89ab6c0-0bd2-4cb5-aca9-71b07c462005\") " pod="openshift-insights/insights-runtime-extractor-ncxvc" Jul 21 02:10:12.533562 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:12.533529 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a89ab6c0-0bd2-4cb5-aca9-71b07c462005-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ncxvc\" (UID: \"a89ab6c0-0bd2-4cb5-aca9-71b07c462005\") " pod="openshift-insights/insights-runtime-extractor-ncxvc" Jul 21 02:10:12.533710 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:10:12.533675 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 02:10:12.533710 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:12.533685 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a89ab6c0-0bd2-4cb5-aca9-71b07c462005-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ncxvc\" (UID: \"a89ab6c0-0bd2-4cb5-aca9-71b07c462005\") " pod="openshift-insights/insights-runtime-extractor-ncxvc" Jul 21 02:10:12.533795 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:10:12.533741 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a89ab6c0-0bd2-4cb5-aca9-71b07c462005-insights-runtime-extractor-tls podName:a89ab6c0-0bd2-4cb5-aca9-71b07c462005 nodeName:}" failed. No retries permitted until 2026-07-21 02:10:13.533726612 +0000 UTC m=+47.553220495 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a89ab6c0-0bd2-4cb5-aca9-71b07c462005-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ncxvc" (UID: "a89ab6c0-0bd2-4cb5-aca9-71b07c462005") : secret "insights-runtime-extractor-tls" not found Jul 21 02:10:12.533795 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:10:12.533747 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 21 02:10:12.533795 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:10:12.533793 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a89ab6c0-0bd2-4cb5-aca9-71b07c462005-kube-rbac-proxy-cm podName:a89ab6c0-0bd2-4cb5-aca9-71b07c462005 nodeName:}" failed. No retries permitted until 2026-07-21 02:10:13.533780404 +0000 UTC m=+47.553274304 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a89ab6c0-0bd2-4cb5-aca9-71b07c462005-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ncxvc" (UID: "a89ab6c0-0bd2-4cb5-aca9-71b07c462005") : configmap "kube-rbac-proxy" not found Jul 21 02:10:13.541652 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:13.541607 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a89ab6c0-0bd2-4cb5-aca9-71b07c462005-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ncxvc\" (UID: \"a89ab6c0-0bd2-4cb5-aca9-71b07c462005\") " pod="openshift-insights/insights-runtime-extractor-ncxvc" Jul 21 02:10:13.542024 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:13.541729 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a89ab6c0-0bd2-4cb5-aca9-71b07c462005-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ncxvc\" (UID: \"a89ab6c0-0bd2-4cb5-aca9-71b07c462005\") " pod="openshift-insights/insights-runtime-extractor-ncxvc" Jul 21 02:10:13.542024 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:10:13.541777 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 02:10:13.542024 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:10:13.541800 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 21 02:10:13.542024 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:10:13.541846 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a89ab6c0-0bd2-4cb5-aca9-71b07c462005-insights-runtime-extractor-tls podName:a89ab6c0-0bd2-4cb5-aca9-71b07c462005 nodeName:}" failed. No retries permitted until 2026-07-21 02:10:15.541832983 +0000 UTC m=+49.561326869 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a89ab6c0-0bd2-4cb5-aca9-71b07c462005-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ncxvc" (UID: "a89ab6c0-0bd2-4cb5-aca9-71b07c462005") : secret "insights-runtime-extractor-tls" not found Jul 21 02:10:13.542024 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:10:13.541863 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a89ab6c0-0bd2-4cb5-aca9-71b07c462005-kube-rbac-proxy-cm podName:a89ab6c0-0bd2-4cb5-aca9-71b07c462005 nodeName:}" failed. No retries permitted until 2026-07-21 02:10:15.541856903 +0000 UTC m=+49.561350789 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a89ab6c0-0bd2-4cb5-aca9-71b07c462005-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ncxvc" (UID: "a89ab6c0-0bd2-4cb5-aca9-71b07c462005") : configmap "kube-rbac-proxy" not found Jul 21 02:10:13.552944 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:13.552922 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8d69j_c8e442c1-e98a-4af5-9274-5ab79f671aef/dns-node-resolver/0.log" Jul 21 02:10:14.749792 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:14.749764 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-b9rfh_83fe72d8-5aa8-4232-9317-3e0d06bc9535/node-ca/0.log" Jul 21 02:10:15.552789 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:15.552765 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pc4fb_224189a4-715f-4007-8d45-832188c1f1d1/serve-healthcheck-canary/0.log" Jul 21 02:10:15.554954 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:15.554934 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a89ab6c0-0bd2-4cb5-aca9-71b07c462005-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ncxvc\" (UID: \"a89ab6c0-0bd2-4cb5-aca9-71b07c462005\") " pod="openshift-insights/insights-runtime-extractor-ncxvc" Jul 21 02:10:15.555045 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:15.555032 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a89ab6c0-0bd2-4cb5-aca9-71b07c462005-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ncxvc\" (UID: \"a89ab6c0-0bd2-4cb5-aca9-71b07c462005\") " pod="openshift-insights/insights-runtime-extractor-ncxvc" Jul 21 02:10:15.555087 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:10:15.555071 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 02:10:15.555120 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:10:15.555112 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 21 02:10:15.555150 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:10:15.555129 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a89ab6c0-0bd2-4cb5-aca9-71b07c462005-insights-runtime-extractor-tls podName:a89ab6c0-0bd2-4cb5-aca9-71b07c462005 nodeName:}" failed. No retries permitted until 2026-07-21 02:10:19.555115374 +0000 UTC m=+53.574609261 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a89ab6c0-0bd2-4cb5-aca9-71b07c462005-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ncxvc" (UID: "a89ab6c0-0bd2-4cb5-aca9-71b07c462005") : secret "insights-runtime-extractor-tls" not found Jul 21 02:10:15.555188 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:10:15.555154 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a89ab6c0-0bd2-4cb5-aca9-71b07c462005-kube-rbac-proxy-cm podName:a89ab6c0-0bd2-4cb5-aca9-71b07c462005 nodeName:}" failed. No retries permitted until 2026-07-21 02:10:19.555144397 +0000 UTC m=+53.574638280 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a89ab6c0-0bd2-4cb5-aca9-71b07c462005-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ncxvc" (UID: "a89ab6c0-0bd2-4cb5-aca9-71b07c462005") : configmap "kube-rbac-proxy" not found Jul 21 02:10:19.084614 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:19.084574 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5afc6d56-b4d0-46f3-958d-ea3b873ee718-registry-tls\") pod \"image-registry-7847d77f6d-9wndp\" (UID: \"5afc6d56-b4d0-46f3-958d-ea3b873ee718\") " pod="openshift-image-registry/image-registry-7847d77f6d-9wndp" Jul 21 02:10:19.084999 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:10:19.084717 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 02:10:19.084999 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:10:19.084729 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7847d77f6d-9wndp: secret "image-registry-tls" not found Jul 21 02:10:19.084999 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:10:19.084778 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5afc6d56-b4d0-46f3-958d-ea3b873ee718-registry-tls podName:5afc6d56-b4d0-46f3-958d-ea3b873ee718 nodeName:}" failed. No retries permitted until 2026-07-21 02:10:35.084763765 +0000 UTC m=+69.104257647 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5afc6d56-b4d0-46f3-958d-ea3b873ee718-registry-tls") pod "image-registry-7847d77f6d-9wndp" (UID: "5afc6d56-b4d0-46f3-958d-ea3b873ee718") : secret "image-registry-tls" not found Jul 21 02:10:19.185624 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:19.185587 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f7a19bce-85cf-4f26-aafa-977786cbfc4d-metrics-tls\") pod \"dns-default-24k47\" (UID: \"f7a19bce-85cf-4f26-aafa-977786cbfc4d\") " pod="openshift-dns/dns-default-24k47" Jul 21 02:10:19.185791 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:10:19.185731 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 02:10:19.185830 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:10:19.185804 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7a19bce-85cf-4f26-aafa-977786cbfc4d-metrics-tls podName:f7a19bce-85cf-4f26-aafa-977786cbfc4d nodeName:}" failed. No retries permitted until 2026-07-21 02:10:35.18578756 +0000 UTC m=+69.205281445 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f7a19bce-85cf-4f26-aafa-977786cbfc4d-metrics-tls") pod "dns-default-24k47" (UID: "f7a19bce-85cf-4f26-aafa-977786cbfc4d") : secret "dns-default-metrics-tls" not found Jul 21 02:10:19.588714 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:19.588676 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a89ab6c0-0bd2-4cb5-aca9-71b07c462005-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ncxvc\" (UID: \"a89ab6c0-0bd2-4cb5-aca9-71b07c462005\") " pod="openshift-insights/insights-runtime-extractor-ncxvc" Jul 21 02:10:19.588991 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:19.588725 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a89ab6c0-0bd2-4cb5-aca9-71b07c462005-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ncxvc\" (UID: \"a89ab6c0-0bd2-4cb5-aca9-71b07c462005\") " pod="openshift-insights/insights-runtime-extractor-ncxvc" Jul 21 02:10:19.588991 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:10:19.588904 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 02:10:19.588991 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:10:19.588967 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a89ab6c0-0bd2-4cb5-aca9-71b07c462005-insights-runtime-extractor-tls podName:a89ab6c0-0bd2-4cb5-aca9-71b07c462005 nodeName:}" failed. No retries permitted until 2026-07-21 02:10:27.588951668 +0000 UTC m=+61.608445550 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a89ab6c0-0bd2-4cb5-aca9-71b07c462005-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ncxvc" (UID: "a89ab6c0-0bd2-4cb5-aca9-71b07c462005") : secret "insights-runtime-extractor-tls" not found Jul 21 02:10:19.589367 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:19.589345 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a89ab6c0-0bd2-4cb5-aca9-71b07c462005-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ncxvc\" (UID: \"a89ab6c0-0bd2-4cb5-aca9-71b07c462005\") " pod="openshift-insights/insights-runtime-extractor-ncxvc" Jul 21 02:10:27.646471 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:27.646415 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a89ab6c0-0bd2-4cb5-aca9-71b07c462005-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ncxvc\" (UID: \"a89ab6c0-0bd2-4cb5-aca9-71b07c462005\") " pod="openshift-insights/insights-runtime-extractor-ncxvc" Jul 21 02:10:27.648740 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:27.648714 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a89ab6c0-0bd2-4cb5-aca9-71b07c462005-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ncxvc\" (UID: \"a89ab6c0-0bd2-4cb5-aca9-71b07c462005\") " pod="openshift-insights/insights-runtime-extractor-ncxvc" Jul 21 02:10:27.731951 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:27.731915 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ncxvc" Jul 21 02:10:27.850599 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:27.850570 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ncxvc"] Jul 21 02:10:27.853751 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:10:27.853720 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda89ab6c0_0bd2_4cb5_aca9_71b07c462005.slice/crio-bb5972ff7129f2d27ba2145c92c4c1e1b8d5ab28f511d75d4bf2bc2532444fd0 WatchSource:0}: Error finding container bb5972ff7129f2d27ba2145c92c4c1e1b8d5ab28f511d75d4bf2bc2532444fd0: Status 404 returned error can't find the container with id bb5972ff7129f2d27ba2145c92c4c1e1b8d5ab28f511d75d4bf2bc2532444fd0 Jul 21 02:10:27.951319 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:27.951290 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ncxvc" event={"ID":"a89ab6c0-0bd2-4cb5-aca9-71b07c462005","Type":"ContainerStarted","Data":"bb5972ff7129f2d27ba2145c92c4c1e1b8d5ab28f511d75d4bf2bc2532444fd0"} Jul 21 02:10:29.883086 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:29.883045 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qv9kr" Jul 21 02:10:32.965956 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:32.965921 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ncxvc" event={"ID":"a89ab6c0-0bd2-4cb5-aca9-71b07c462005","Type":"ContainerStarted","Data":"7a262c54c710ac17713f6d1675b55fb3985ecb407589576748ae88976f02dda3"} Jul 21 02:10:33.092854 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:33.092811 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03e524b0-40bd-46b4-ac8f-eb346e94a7dc-metrics-certs\") pod \"network-metrics-daemon-8v67q\" (UID: \"03e524b0-40bd-46b4-ac8f-eb346e94a7dc\") " pod="openshift-multus/network-metrics-daemon-8v67q" Jul 21 02:10:33.095933 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:33.095908 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 21 02:10:33.106112 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:33.106085 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03e524b0-40bd-46b4-ac8f-eb346e94a7dc-metrics-certs\") pod \"network-metrics-daemon-8v67q\" (UID: \"03e524b0-40bd-46b4-ac8f-eb346e94a7dc\") " pod="openshift-multus/network-metrics-daemon-8v67q" Jul 21 02:10:33.193505 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:33.193466 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4bee3776-d7db-4be5-8e76-337ee2b89187-original-pull-secret\") pod \"global-pull-secret-syncer-ljdcc\" (UID: \"4bee3776-d7db-4be5-8e76-337ee2b89187\") " pod="kube-system/global-pull-secret-syncer-ljdcc" Jul 21 02:10:33.196876 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:33.196832 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 21 02:10:33.207134 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:33.207105 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4bee3776-d7db-4be5-8e76-337ee2b89187-original-pull-secret\") pod \"global-pull-secret-syncer-ljdcc\" (UID: \"4bee3776-d7db-4be5-8e76-337ee2b89187\") " pod="kube-system/global-pull-secret-syncer-ljdcc" Jul 21 02:10:33.293590 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:33.293509 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2zscq\"" Jul 21 02:10:33.294023 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:33.293999 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht26w\" (UniqueName: \"kubernetes.io/projected/80deee90-4b47-4cdf-bc87-f9242ad63cec-kube-api-access-ht26w\") pod \"network-check-target-qqnx4\" (UID: \"80deee90-4b47-4cdf-bc87-f9242ad63cec\") " pod="openshift-network-diagnostics/network-check-target-qqnx4" Jul 21 02:10:33.296623 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:33.296603 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 21 02:10:33.299779 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:33.299755 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ljdcc" Jul 21 02:10:33.300826 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:33.300806 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8v67q" Jul 21 02:10:33.306738 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:33.306716 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 21 02:10:33.318367 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:33.318346 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht26w\" (UniqueName: \"kubernetes.io/projected/80deee90-4b47-4cdf-bc87-f9242ad63cec-kube-api-access-ht26w\") pod \"network-check-target-qqnx4\" (UID: \"80deee90-4b47-4cdf-bc87-f9242ad63cec\") " pod="openshift-network-diagnostics/network-check-target-qqnx4" Jul 21 02:10:33.468569 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:33.468535 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ljdcc"] Jul 21 02:10:33.482674 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:33.482564 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8v67q"] Jul 21 02:10:33.489096 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:10:33.489064 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03e524b0_40bd_46b4_ac8f_eb346e94a7dc.slice/crio-6be152c0039bd5a95f5d05dec47baabb3a3e04affc14264003846484339ad24a WatchSource:0}: Error finding container 6be152c0039bd5a95f5d05dec47baabb3a3e04affc14264003846484339ad24a: Status 404 returned error can't find the container with id 6be152c0039bd5a95f5d05dec47baabb3a3e04affc14264003846484339ad24a Jul 21 02:10:33.491351 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:33.491140 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gcvct\"" Jul 21 02:10:33.499767 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:33.499738 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqnx4" Jul 21 02:10:33.638211 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:33.638176 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7455dd7b88-ggvf9"] Jul 21 02:10:33.641402 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:33.641380 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7455dd7b88-ggvf9" Jul 21 02:10:33.644533 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:33.644507 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 21 02:10:33.644656 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:33.644617 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 21 02:10:33.644713 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:33.644651 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 21 02:10:33.645638 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:33.645620 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 21 02:10:33.649727 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:33.649706 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qqnx4"] Jul 21 02:10:33.653677 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:10:33.653649 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80deee90_4b47_4cdf_bc87_f9242ad63cec.slice/crio-4097f86d8342a8408a361f4b285a1a1c17f4a85b63f3cbb1ab9bcdb09d2d06ce WatchSource:0}: Error finding container 4097f86d8342a8408a361f4b285a1a1c17f4a85b63f3cbb1ab9bcdb09d2d06ce: Status 404 returned error can't find the container with id 4097f86d8342a8408a361f4b285a1a1c17f4a85b63f3cbb1ab9bcdb09d2d06ce Jul 21 02:10:33.656114 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:33.656055 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7455dd7b88-ggvf9"] Jul 21 02:10:33.669590 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:33.669560 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7847d77f6d-9wndp"] Jul 21 02:10:33.669823 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:10:33.669801 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-7847d77f6d-9wndp" podUID="5afc6d56-b4d0-46f3-958d-ea3b873ee718" Jul 21 02:10:33.697561 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:33.697527 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/bc35cf29-f72a-49e0-83f8-7d52da643608-klusterlet-config\") pod \"klusterlet-addon-workmgr-7455dd7b88-ggvf9\" (UID: \"bc35cf29-f72a-49e0-83f8-7d52da643608\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7455dd7b88-ggvf9" Jul 21 02:10:33.697704 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:33.697572 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc35cf29-f72a-49e0-83f8-7d52da643608-tmp\") pod \"klusterlet-addon-workmgr-7455dd7b88-ggvf9\" (UID: \"bc35cf29-f72a-49e0-83f8-7d52da643608\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7455dd7b88-ggvf9" Jul 21 02:10:33.697704 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:33.697606 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrbbk\" (UniqueName: \"kubernetes.io/projected/bc35cf29-f72a-49e0-83f8-7d52da643608-kube-api-access-wrbbk\") pod \"klusterlet-addon-workmgr-7455dd7b88-ggvf9\" (UID: \"bc35cf29-f72a-49e0-83f8-7d52da643608\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7455dd7b88-ggvf9" Jul 21 02:10:33.724785 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:33.724755 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7bc6c6857c-7rj6n"] Jul 21 02:10:33.727904 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:33.727880 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bc6c6857c-7rj6n" Jul 21 02:10:33.739880 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:33.739853 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7bc6c6857c-7rj6n"] Jul 21 02:10:33.798384 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:33.798347 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a56f6c6c-6e08-41fc-b8ad-69c9fb0f82e3-trusted-ca\") pod \"image-registry-7bc6c6857c-7rj6n\" (UID: \"a56f6c6c-6e08-41fc-b8ad-69c9fb0f82e3\") " pod="openshift-image-registry/image-registry-7bc6c6857c-7rj6n" Jul 21 02:10:33.798544 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:33.798414 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/bc35cf29-f72a-49e0-83f8-7d52da643608-klusterlet-config\") pod \"klusterlet-addon-workmgr-7455dd7b88-ggvf9\" (UID: \"bc35cf29-f72a-49e0-83f8-7d52da643608\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7455dd7b88-ggvf9" Jul 21 02:10:33.798544 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:33.798452 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrbbk\" (UniqueName: \"kubernetes.io/projected/bc35cf29-f72a-49e0-83f8-7d52da643608-kube-api-access-wrbbk\") pod \"klusterlet-addon-workmgr-7455dd7b88-ggvf9\" (UID: \"bc35cf29-f72a-49e0-83f8-7d52da643608\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7455dd7b88-ggvf9" Jul 21 02:10:33.798544 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:33.798488 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a56f6c6c-6e08-41fc-b8ad-69c9fb0f82e3-installation-pull-secrets\") pod \"image-registry-7bc6c6857c-7rj6n\" (UID: \"a56f6c6c-6e08-41fc-b8ad-69c9fb0f82e3\") " pod="openshift-image-registry/image-registry-7bc6c6857c-7rj6n" Jul 21 02:10:33.798544 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:33.798542 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a56f6c6c-6e08-41fc-b8ad-69c9fb0f82e3-bound-sa-token\") pod \"image-registry-7bc6c6857c-7rj6n\" (UID: \"a56f6c6c-6e08-41fc-b8ad-69c9fb0f82e3\") " pod="openshift-image-registry/image-registry-7bc6c6857c-7rj6n" Jul 21 02:10:33.798758 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:33.798569 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a56f6c6c-6e08-41fc-b8ad-69c9fb0f82e3-ca-trust-extracted\") pod \"image-registry-7bc6c6857c-7rj6n\" (UID: \"a56f6c6c-6e08-41fc-b8ad-69c9fb0f82e3\") " pod="openshift-image-registry/image-registry-7bc6c6857c-7rj6n" Jul 21 02:10:33.798758 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:33.798594 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc35cf29-f72a-49e0-83f8-7d52da643608-tmp\") pod \"klusterlet-addon-workmgr-7455dd7b88-ggvf9\" (UID: \"bc35cf29-f72a-49e0-83f8-7d52da643608\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7455dd7b88-ggvf9" Jul 21 02:10:33.798758 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:33.798657 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnp88\" (UniqueName: \"kubernetes.io/projected/a56f6c6c-6e08-41fc-b8ad-69c9fb0f82e3-kube-api-access-rnp88\") pod \"image-registry-7bc6c6857c-7rj6n\" (UID: \"a56f6c6c-6e08-41fc-b8ad-69c9fb0f82e3\") " pod="openshift-image-registry/image-registry-7bc6c6857c-7rj6n" Jul 21 02:10:33.798758 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:33.798699 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a56f6c6c-6e08-41fc-b8ad-69c9fb0f82e3-registry-certificates\") pod \"image-registry-7bc6c6857c-7rj6n\" (UID: \"a56f6c6c-6e08-41fc-b8ad-69c9fb0f82e3\") " pod="openshift-image-registry/image-registry-7bc6c6857c-7rj6n" Jul 21 02:10:33.798758 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:33.798737 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a56f6c6c-6e08-41fc-b8ad-69c9fb0f82e3-registry-tls\") pod \"image-registry-7bc6c6857c-7rj6n\" (UID: \"a56f6c6c-6e08-41fc-b8ad-69c9fb0f82e3\") " pod="openshift-image-registry/image-registry-7bc6c6857c-7rj6n" Jul 21 02:10:33.798962 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:33.798767 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a56f6c6c-6e08-41fc-b8ad-69c9fb0f82e3-image-registry-private-configuration\") pod \"image-registry-7bc6c6857c-7rj6n\" (UID: \"a56f6c6c-6e08-41fc-b8ad-69c9fb0f82e3\") " pod="openshift-image-registry/image-registry-7bc6c6857c-7rj6n" Jul 21 02:10:33.798996 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:33.798982 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc35cf29-f72a-49e0-83f8-7d52da643608-tmp\") pod \"klusterlet-addon-workmgr-7455dd7b88-ggvf9\" (UID: \"bc35cf29-f72a-49e0-83f8-7d52da643608\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7455dd7b88-ggvf9" Jul 21 02:10:33.801495 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:33.801470 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/bc35cf29-f72a-49e0-83f8-7d52da643608-klusterlet-config\") pod \"klusterlet-addon-workmgr-7455dd7b88-ggvf9\" (UID: \"bc35cf29-f72a-49e0-83f8-7d52da643608\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7455dd7b88-ggvf9" Jul 21 02:10:33.830086 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:33.829983 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrbbk\" (UniqueName: \"kubernetes.io/projected/bc35cf29-f72a-49e0-83f8-7d52da643608-kube-api-access-wrbbk\") pod \"klusterlet-addon-workmgr-7455dd7b88-ggvf9\" (UID: \"bc35cf29-f72a-49e0-83f8-7d52da643608\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7455dd7b88-ggvf9" Jul 21 02:10:33.900161 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:33.899242 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnp88\" (UniqueName: \"kubernetes.io/projected/a56f6c6c-6e08-41fc-b8ad-69c9fb0f82e3-kube-api-access-rnp88\") pod \"image-registry-7bc6c6857c-7rj6n\" (UID: \"a56f6c6c-6e08-41fc-b8ad-69c9fb0f82e3\") " pod="openshift-image-registry/image-registry-7bc6c6857c-7rj6n" Jul 21 02:10:33.900161 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:33.899300 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a56f6c6c-6e08-41fc-b8ad-69c9fb0f82e3-registry-certificates\") pod \"image-registry-7bc6c6857c-7rj6n\" (UID: \"a56f6c6c-6e08-41fc-b8ad-69c9fb0f82e3\") " pod="openshift-image-registry/image-registry-7bc6c6857c-7rj6n" Jul 21 02:10:33.900161 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:33.899331 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a56f6c6c-6e08-41fc-b8ad-69c9fb0f82e3-registry-tls\") pod \"image-registry-7bc6c6857c-7rj6n\" (UID: \"a56f6c6c-6e08-41fc-b8ad-69c9fb0f82e3\") " pod="openshift-image-registry/image-registry-7bc6c6857c-7rj6n" Jul 21 02:10:33.900161 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:33.899356 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a56f6c6c-6e08-41fc-b8ad-69c9fb0f82e3-image-registry-private-configuration\") pod \"image-registry-7bc6c6857c-7rj6n\" (UID: \"a56f6c6c-6e08-41fc-b8ad-69c9fb0f82e3\") " pod="openshift-image-registry/image-registry-7bc6c6857c-7rj6n" Jul 21 02:10:33.900161 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:33.899386 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a56f6c6c-6e08-41fc-b8ad-69c9fb0f82e3-trusted-ca\") pod \"image-registry-7bc6c6857c-7rj6n\" (UID: \"a56f6c6c-6e08-41fc-b8ad-69c9fb0f82e3\") " pod="openshift-image-registry/image-registry-7bc6c6857c-7rj6n" Jul 21 02:10:33.900161 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:33.899439 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a56f6c6c-6e08-41fc-b8ad-69c9fb0f82e3-installation-pull-secrets\") pod \"image-registry-7bc6c6857c-7rj6n\" (UID: \"a56f6c6c-6e08-41fc-b8ad-69c9fb0f82e3\") " pod="openshift-image-registry/image-registry-7bc6c6857c-7rj6n" Jul 21 02:10:33.900161 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:33.899486 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a56f6c6c-6e08-41fc-b8ad-69c9fb0f82e3-bound-sa-token\") pod \"image-registry-7bc6c6857c-7rj6n\" (UID: \"a56f6c6c-6e08-41fc-b8ad-69c9fb0f82e3\") " pod="openshift-image-registry/image-registry-7bc6c6857c-7rj6n" Jul 21 02:10:33.900161 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:33.899510 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a56f6c6c-6e08-41fc-b8ad-69c9fb0f82e3-ca-trust-extracted\") pod \"image-registry-7bc6c6857c-7rj6n\" (UID: \"a56f6c6c-6e08-41fc-b8ad-69c9fb0f82e3\") " pod="openshift-image-registry/image-registry-7bc6c6857c-7rj6n" Jul 21 02:10:33.900161 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:33.899886 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a56f6c6c-6e08-41fc-b8ad-69c9fb0f82e3-ca-trust-extracted\") pod \"image-registry-7bc6c6857c-7rj6n\" (UID: \"a56f6c6c-6e08-41fc-b8ad-69c9fb0f82e3\") " pod="openshift-image-registry/image-registry-7bc6c6857c-7rj6n" Jul 21 02:10:33.900757 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:33.900207 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a56f6c6c-6e08-41fc-b8ad-69c9fb0f82e3-registry-certificates\") pod \"image-registry-7bc6c6857c-7rj6n\" (UID: \"a56f6c6c-6e08-41fc-b8ad-69c9fb0f82e3\") " pod="openshift-image-registry/image-registry-7bc6c6857c-7rj6n" Jul 21 02:10:33.900757 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:33.900723 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a56f6c6c-6e08-41fc-b8ad-69c9fb0f82e3-trusted-ca\") pod \"image-registry-7bc6c6857c-7rj6n\" (UID: \"a56f6c6c-6e08-41fc-b8ad-69c9fb0f82e3\") " pod="openshift-image-registry/image-registry-7bc6c6857c-7rj6n" Jul 21 02:10:33.902536 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:33.902509 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a56f6c6c-6e08-41fc-b8ad-69c9fb0f82e3-image-registry-private-configuration\") pod \"image-registry-7bc6c6857c-7rj6n\" (UID: \"a56f6c6c-6e08-41fc-b8ad-69c9fb0f82e3\") " pod="openshift-image-registry/image-registry-7bc6c6857c-7rj6n" Jul 21 02:10:33.903283 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:33.903223 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a56f6c6c-6e08-41fc-b8ad-69c9fb0f82e3-registry-tls\") pod \"image-registry-7bc6c6857c-7rj6n\" (UID: \"a56f6c6c-6e08-41fc-b8ad-69c9fb0f82e3\") " pod="openshift-image-registry/image-registry-7bc6c6857c-7rj6n" Jul 21 02:10:33.903526 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:33.903481 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a56f6c6c-6e08-41fc-b8ad-69c9fb0f82e3-installation-pull-secrets\") pod \"image-registry-7bc6c6857c-7rj6n\" (UID: \"a56f6c6c-6e08-41fc-b8ad-69c9fb0f82e3\") " pod="openshift-image-registry/image-registry-7bc6c6857c-7rj6n" Jul 21 02:10:33.911025 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:33.910989 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnp88\" (UniqueName: \"kubernetes.io/projected/a56f6c6c-6e08-41fc-b8ad-69c9fb0f82e3-kube-api-access-rnp88\") pod \"image-registry-7bc6c6857c-7rj6n\" (UID: \"a56f6c6c-6e08-41fc-b8ad-69c9fb0f82e3\") " pod="openshift-image-registry/image-registry-7bc6c6857c-7rj6n" Jul 21 02:10:33.913171 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:33.913100 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a56f6c6c-6e08-41fc-b8ad-69c9fb0f82e3-bound-sa-token\") pod \"image-registry-7bc6c6857c-7rj6n\" (UID: \"a56f6c6c-6e08-41fc-b8ad-69c9fb0f82e3\") " pod="openshift-image-registry/image-registry-7bc6c6857c-7rj6n" Jul 21 02:10:33.952979 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:33.952940 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7455dd7b88-ggvf9" Jul 21 02:10:33.972203 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:33.972163 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qqnx4" event={"ID":"80deee90-4b47-4cdf-bc87-f9242ad63cec","Type":"ContainerStarted","Data":"4097f86d8342a8408a361f4b285a1a1c17f4a85b63f3cbb1ab9bcdb09d2d06ce"} Jul 21 02:10:33.974465 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:33.974431 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8v67q" event={"ID":"03e524b0-40bd-46b4-ac8f-eb346e94a7dc","Type":"ContainerStarted","Data":"6be152c0039bd5a95f5d05dec47baabb3a3e04affc14264003846484339ad24a"} Jul 21 02:10:33.976172 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:33.976151 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7847d77f6d-9wndp" Jul 21 02:10:33.976496 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:33.976474 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ljdcc" event={"ID":"4bee3776-d7db-4be5-8e76-337ee2b89187","Type":"ContainerStarted","Data":"b0bc3feb20061fdd77e27d17800b8577bed65b8d5fbaa94a040b9437bdcd75c6"} Jul 21 02:10:33.986047 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:33.985600 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7847d77f6d-9wndp" Jul 21 02:10:34.003286 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:34.000157 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5afc6d56-b4d0-46f3-958d-ea3b873ee718-bound-sa-token\") pod \"5afc6d56-b4d0-46f3-958d-ea3b873ee718\" (UID: \"5afc6d56-b4d0-46f3-958d-ea3b873ee718\") " Jul 21 02:10:34.003286 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:34.000205 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2kng9\" (UniqueName: \"kubernetes.io/projected/5afc6d56-b4d0-46f3-958d-ea3b873ee718-kube-api-access-2kng9\") pod \"5afc6d56-b4d0-46f3-958d-ea3b873ee718\" (UID: \"5afc6d56-b4d0-46f3-958d-ea3b873ee718\") " Jul 21 02:10:34.003286 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:34.000239 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5afc6d56-b4d0-46f3-958d-ea3b873ee718-image-registry-private-configuration\") pod \"5afc6d56-b4d0-46f3-958d-ea3b873ee718\" (UID: \"5afc6d56-b4d0-46f3-958d-ea3b873ee718\") " Jul 21 02:10:34.003286 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:34.000298 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5afc6d56-b4d0-46f3-958d-ea3b873ee718-ca-trust-extracted\") pod \"5afc6d56-b4d0-46f3-958d-ea3b873ee718\" (UID: \"5afc6d56-b4d0-46f3-958d-ea3b873ee718\") " Jul 21 02:10:34.003286 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:34.000345 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5afc6d56-b4d0-46f3-958d-ea3b873ee718-registry-certificates\") pod \"5afc6d56-b4d0-46f3-958d-ea3b873ee718\" (UID: \"5afc6d56-b4d0-46f3-958d-ea3b873ee718\") " Jul 21 02:10:34.003286 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:34.000375 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5afc6d56-b4d0-46f3-958d-ea3b873ee718-trusted-ca\") pod \"5afc6d56-b4d0-46f3-958d-ea3b873ee718\" (UID: \"5afc6d56-b4d0-46f3-958d-ea3b873ee718\") " Jul 21 02:10:34.003286 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:34.000423 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5afc6d56-b4d0-46f3-958d-ea3b873ee718-installation-pull-secrets\") pod \"5afc6d56-b4d0-46f3-958d-ea3b873ee718\" (UID: \"5afc6d56-b4d0-46f3-958d-ea3b873ee718\") " Jul 21 02:10:34.003286 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:34.002048 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5afc6d56-b4d0-46f3-958d-ea3b873ee718-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "5afc6d56-b4d0-46f3-958d-ea3b873ee718" (UID: "5afc6d56-b4d0-46f3-958d-ea3b873ee718"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 02:10:34.003286 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:34.002374 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5afc6d56-b4d0-46f3-958d-ea3b873ee718-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "5afc6d56-b4d0-46f3-958d-ea3b873ee718" (UID: "5afc6d56-b4d0-46f3-958d-ea3b873ee718"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 02:10:34.003286 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:34.002257 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5afc6d56-b4d0-46f3-958d-ea3b873ee718-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "5afc6d56-b4d0-46f3-958d-ea3b873ee718" (UID: "5afc6d56-b4d0-46f3-958d-ea3b873ee718"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 02:10:34.008238 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:34.006609 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5afc6d56-b4d0-46f3-958d-ea3b873ee718-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "5afc6d56-b4d0-46f3-958d-ea3b873ee718" (UID: "5afc6d56-b4d0-46f3-958d-ea3b873ee718"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 02:10:34.008238 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:34.006613 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5afc6d56-b4d0-46f3-958d-ea3b873ee718-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "5afc6d56-b4d0-46f3-958d-ea3b873ee718" (UID: "5afc6d56-b4d0-46f3-958d-ea3b873ee718"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 02:10:34.008238 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:34.006721 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5afc6d56-b4d0-46f3-958d-ea3b873ee718-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "5afc6d56-b4d0-46f3-958d-ea3b873ee718" (UID: "5afc6d56-b4d0-46f3-958d-ea3b873ee718"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 02:10:34.008506 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:34.008478 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5afc6d56-b4d0-46f3-958d-ea3b873ee718-kube-api-access-2kng9" (OuterVolumeSpecName: "kube-api-access-2kng9") pod "5afc6d56-b4d0-46f3-958d-ea3b873ee718" (UID: "5afc6d56-b4d0-46f3-958d-ea3b873ee718"). InnerVolumeSpecName "kube-api-access-2kng9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 02:10:34.041325 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:34.041295 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6qdf5\"" Jul 21 02:10:34.049784 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:34.049754 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bc6c6857c-7rj6n" Jul 21 02:10:34.101655 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:34.101569 2574 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5afc6d56-b4d0-46f3-958d-ea3b873ee718-registry-certificates\") on node \"ip-10-0-141-37.ec2.internal\" DevicePath \"\"" Jul 21 02:10:34.101655 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:34.101605 2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5afc6d56-b4d0-46f3-958d-ea3b873ee718-trusted-ca\") on node \"ip-10-0-141-37.ec2.internal\" DevicePath \"\"" Jul 21 02:10:34.101655 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:34.101622 2574 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5afc6d56-b4d0-46f3-958d-ea3b873ee718-installation-pull-secrets\") on node \"ip-10-0-141-37.ec2.internal\" DevicePath \"\"" Jul 21 02:10:34.101655 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:34.101635 2574 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5afc6d56-b4d0-46f3-958d-ea3b873ee718-bound-sa-token\") on node \"ip-10-0-141-37.ec2.internal\" DevicePath \"\"" Jul 21 02:10:34.101655 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:34.101648 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kng9\" (UniqueName: \"kubernetes.io/projected/5afc6d56-b4d0-46f3-958d-ea3b873ee718-kube-api-access-2kng9\") on node \"ip-10-0-141-37.ec2.internal\" DevicePath \"\"" Jul 21 02:10:34.102006 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:34.101661 2574 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5afc6d56-b4d0-46f3-958d-ea3b873ee718-image-registry-private-configuration\") on node \"ip-10-0-141-37.ec2.internal\" DevicePath \"\"" Jul 21 02:10:34.102006 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:34.101675 2574 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5afc6d56-b4d0-46f3-958d-ea3b873ee718-ca-trust-extracted\") on node \"ip-10-0-141-37.ec2.internal\" DevicePath \"\"" Jul 21 02:10:34.129614 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:34.129583 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7455dd7b88-ggvf9"] Jul 21 02:10:34.136613 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:10:34.136578 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc35cf29_f72a_49e0_83f8_7d52da643608.slice/crio-df8236a2e8ef05755f08ba8dabf8d2060e9efeeca0311359ba9260cfdf6584f1 WatchSource:0}: Error finding container df8236a2e8ef05755f08ba8dabf8d2060e9efeeca0311359ba9260cfdf6584f1: Status 404 returned error can't find the container with id df8236a2e8ef05755f08ba8dabf8d2060e9efeeca0311359ba9260cfdf6584f1 Jul 21 02:10:34.235984 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:34.235940 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7bc6c6857c-7rj6n"] Jul 21 02:10:34.983895 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:34.983394 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7847d77f6d-9wndp" Jul 21 02:10:34.983895 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:34.983384 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7455dd7b88-ggvf9" event={"ID":"bc35cf29-f72a-49e0-83f8-7d52da643608","Type":"ContainerStarted","Data":"df8236a2e8ef05755f08ba8dabf8d2060e9efeeca0311359ba9260cfdf6584f1"} Jul 21 02:10:35.020497 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:35.020440 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7847d77f6d-9wndp"] Jul 21 02:10:35.024492 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:35.024464 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7847d77f6d-9wndp"] Jul 21 02:10:35.110207 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:35.110167 2574 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5afc6d56-b4d0-46f3-958d-ea3b873ee718-registry-tls\") on node \"ip-10-0-141-37.ec2.internal\" DevicePath \"\"" Jul 21 02:10:35.211546 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:35.211399 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f7a19bce-85cf-4f26-aafa-977786cbfc4d-metrics-tls\") pod \"dns-default-24k47\" (UID: \"f7a19bce-85cf-4f26-aafa-977786cbfc4d\") " pod="openshift-dns/dns-default-24k47" Jul 21 02:10:35.216242 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:35.216215 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f7a19bce-85cf-4f26-aafa-977786cbfc4d-metrics-tls\") pod \"dns-default-24k47\" (UID: \"f7a19bce-85cf-4f26-aafa-977786cbfc4d\") " pod="openshift-dns/dns-default-24k47" Jul 21 02:10:35.419380 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:35.419303 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lx8r6\"" Jul 21 02:10:35.427756 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:35.427555 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-24k47" Jul 21 02:10:35.914359 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:10:35.914322 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda56f6c6c_6e08_41fc_b8ad_69c9fb0f82e3.slice/crio-4f4ca56ddf31582ec85ff48b4e10f96305ab80e51011fcdd8a7612ac9db4e1cf WatchSource:0}: Error finding container 4f4ca56ddf31582ec85ff48b4e10f96305ab80e51011fcdd8a7612ac9db4e1cf: Status 404 returned error can't find the container with id 4f4ca56ddf31582ec85ff48b4e10f96305ab80e51011fcdd8a7612ac9db4e1cf Jul 21 02:10:35.988483 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:35.988448 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7bc6c6857c-7rj6n" event={"ID":"a56f6c6c-6e08-41fc-b8ad-69c9fb0f82e3","Type":"ContainerStarted","Data":"4f4ca56ddf31582ec85ff48b4e10f96305ab80e51011fcdd8a7612ac9db4e1cf"} Jul 21 02:10:36.483111 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:36.483080 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5afc6d56-b4d0-46f3-958d-ea3b873ee718" path="/var/lib/kubelet/pods/5afc6d56-b4d0-46f3-958d-ea3b873ee718/volumes" Jul 21 02:10:40.697954 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:40.697922 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-24k47"] Jul 21 02:10:40.718732 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:40.718705 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-7hdf7"] Jul 21 02:10:40.721072 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:40.721053 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-7hdf7" Jul 21 02:10:40.723499 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:40.723476 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-bkvsq\"" Jul 21 02:10:40.723604 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:40.723543 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 21 02:10:40.729314 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:40.729243 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-7hdf7"] Jul 21 02:10:40.755045 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:40.754972 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/879894ab-6a37-4e4e-8427-2d199eaaa51e-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-7hdf7\" (UID: \"879894ab-6a37-4e4e-8427-2d199eaaa51e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-7hdf7" Jul 21 02:10:40.855977 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:40.855941 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/879894ab-6a37-4e4e-8427-2d199eaaa51e-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-7hdf7\" (UID: \"879894ab-6a37-4e4e-8427-2d199eaaa51e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-7hdf7" Jul 21 02:10:40.856146 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:10:40.856097 2574 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 21 02:10:40.856197 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:10:40.856185 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/879894ab-6a37-4e4e-8427-2d199eaaa51e-tls-certificates podName:879894ab-6a37-4e4e-8427-2d199eaaa51e nodeName:}" failed. No retries permitted until 2026-07-21 02:10:41.356164659 +0000 UTC m=+75.375658557 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/879894ab-6a37-4e4e-8427-2d199eaaa51e-tls-certificates") pod "prometheus-operator-admission-webhook-85574b5f49-7hdf7" (UID: "879894ab-6a37-4e4e-8427-2d199eaaa51e") : secret "prometheus-operator-admission-webhook-tls" not found Jul 21 02:10:41.360286 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:41.360234 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/879894ab-6a37-4e4e-8427-2d199eaaa51e-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-7hdf7\" (UID: \"879894ab-6a37-4e4e-8427-2d199eaaa51e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-7hdf7" Jul 21 02:10:41.362916 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:41.362886 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/879894ab-6a37-4e4e-8427-2d199eaaa51e-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-7hdf7\" (UID: \"879894ab-6a37-4e4e-8427-2d199eaaa51e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-7hdf7" Jul 21 02:10:41.631828 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:41.631743 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-7hdf7" Jul 21 02:10:41.681815 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:10:41.681785 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7a19bce_85cf_4f26_aafa_977786cbfc4d.slice/crio-845a70ede104e5bf5ede07c2c83911b3b19744cda8882ef41b85f597a2f59643 WatchSource:0}: Error finding container 845a70ede104e5bf5ede07c2c83911b3b19744cda8882ef41b85f597a2f59643: Status 404 returned error can't find the container with id 845a70ede104e5bf5ede07c2c83911b3b19744cda8882ef41b85f597a2f59643 Jul 21 02:10:41.867062 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:41.865864 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-7hdf7"] Jul 21 02:10:41.877463 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:10:41.876403 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod879894ab_6a37_4e4e_8427_2d199eaaa51e.slice/crio-a93a7264970e0ad9f61e0e18c1bb16aeb0da9bf4c07a161c10bc36420a4843e1 WatchSource:0}: Error finding container a93a7264970e0ad9f61e0e18c1bb16aeb0da9bf4c07a161c10bc36420a4843e1: Status 404 returned error can't find the container with id a93a7264970e0ad9f61e0e18c1bb16aeb0da9bf4c07a161c10bc36420a4843e1 Jul 21 02:10:42.008412 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:42.008375 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8v67q" event={"ID":"03e524b0-40bd-46b4-ac8f-eb346e94a7dc","Type":"ContainerStarted","Data":"1cc37c5739b002736ea1643a00b9a705007d0edeaddea53187e1ceff4f3f5c15"} Jul 21 02:10:42.008586 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:42.008437 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8v67q" event={"ID":"03e524b0-40bd-46b4-ac8f-eb346e94a7dc","Type":"ContainerStarted","Data":"79ba37190a44dad57506a4981f31af665917be41c1348d2e0982dbe4d6d51417"} Jul 21 02:10:42.010081 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:42.010050 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7bc6c6857c-7rj6n" event={"ID":"a56f6c6c-6e08-41fc-b8ad-69c9fb0f82e3","Type":"ContainerStarted","Data":"9be385d6573f153eca1dd49101c6e39b6ebf144025be3873fad6200b9dd94c7e"} Jul 21 02:10:42.010216 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:42.010119 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7bc6c6857c-7rj6n" Jul 21 02:10:42.011297 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:42.011261 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-7hdf7" event={"ID":"879894ab-6a37-4e4e-8427-2d199eaaa51e","Type":"ContainerStarted","Data":"a93a7264970e0ad9f61e0e18c1bb16aeb0da9bf4c07a161c10bc36420a4843e1"} Jul 21 02:10:42.012823 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:42.012801 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ljdcc" event={"ID":"4bee3776-d7db-4be5-8e76-337ee2b89187","Type":"ContainerStarted","Data":"2b51815bfeb754b86f49f442cfca71401ae6c6e5514f4658854246e063c84982"} Jul 21 02:10:42.014321 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:42.014298 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qqnx4" event={"ID":"80deee90-4b47-4cdf-bc87-f9242ad63cec","Type":"ContainerStarted","Data":"3b6f7a6da13a507cfc4fa502894d494fb3db22cf8aead25208dfdd8e54d1354f"} Jul 21 02:10:42.014446 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:42.014420 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qqnx4" Jul 21 02:10:42.015904 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:42.015881 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7455dd7b88-ggvf9" event={"ID":"bc35cf29-f72a-49e0-83f8-7d52da643608","Type":"ContainerStarted","Data":"f13c6a88853bcfcfb81e1ed3f6794fb8e539c863f83b8e19619918ea28c75b07"} Jul 21 02:10:42.015994 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:42.015925 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7455dd7b88-ggvf9" Jul 21 02:10:42.017896 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:42.017859 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7455dd7b88-ggvf9" Jul 21 02:10:42.018068 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:42.018041 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ncxvc" event={"ID":"a89ab6c0-0bd2-4cb5-aca9-71b07c462005","Type":"ContainerStarted","Data":"6bae141f555dac7f9401f9734d33bccd1c5f4fb2c10ea92fd482ad8de5ee0461"} Jul 21 02:10:42.019115 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:42.019095 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-24k47" event={"ID":"f7a19bce-85cf-4f26-aafa-977786cbfc4d","Type":"ContainerStarted","Data":"845a70ede104e5bf5ede07c2c83911b3b19744cda8882ef41b85f597a2f59643"} Jul 21 02:10:42.024917 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:42.024879 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-8v67q" podStartSLOduration=65.820290915 podStartE2EDuration="1m14.024866956s" totalImagesPullingTime="8.204576041s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 02:09:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:10:42.024148007 +0000 UTC m=+76.043641913" watchObservedRunningTime="2026-07-21 02:10:42.024866956 +0000 UTC m=+76.044360860" Jul 21 02:10:42.038065 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:42.038025 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-ljdcc" podStartSLOduration=65.818858747 podStartE2EDuration="1m14.03801257s" totalImagesPullingTime="8.219153823s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 02:09:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:10:42.037214946 +0000 UTC m=+76.056708873" watchObservedRunningTime="2026-07-21 02:10:42.03801257 +0000 UTC m=+76.057506473" Jul 21 02:10:42.052751 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:42.052710 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7455dd7b88-ggvf9" podStartSLOduration=1.469625642 podStartE2EDuration="9.052697615s" totalImagesPullingTime="7.583071973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 02:10:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:10:42.052067831 +0000 UTC m=+76.071561743" watchObservedRunningTime="2026-07-21 02:10:42.052697615 +0000 UTC m=+76.072191498" Jul 21 02:10:42.068070 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:42.067898 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qqnx4" podStartSLOduration=67.160117834 podStartE2EDuration="1m14.067876749s" totalImagesPullingTime="6.907758915s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 02:09:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:10:42.066787215 +0000 UTC m=+76.086281145" watchObservedRunningTime="2026-07-21 02:10:42.067876749 +0000 UTC m=+76.087370653" Jul 21 02:10:42.091078 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:42.090986 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7bc6c6857c-7rj6n" podStartSLOduration=9.090967765 podStartE2EDuration="9.090967765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 02:10:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:10:42.090110786 +0000 UTC m=+76.109604711" watchObservedRunningTime="2026-07-21 02:10:42.090967765 +0000 UTC m=+76.110461671" Jul 21 02:10:44.037025 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:44.036928 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-24k47" event={"ID":"f7a19bce-85cf-4f26-aafa-977786cbfc4d","Type":"ContainerStarted","Data":"2b2f4bd2f994b4d9289a078296b73e20c25a8131d7f0dd6f60534d9b25461af5"} Jul 21 02:10:44.037474 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:44.037026 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-24k47" event={"ID":"f7a19bce-85cf-4f26-aafa-977786cbfc4d","Type":"ContainerStarted","Data":"1dd0db328b6769cdb76374144c8341ecb59973ccbb1ba175e95a4bcc33dc0311"} Jul 21 02:10:44.037474 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:44.037057 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-24k47" Jul 21 02:10:44.038678 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:44.038633 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-7hdf7" event={"ID":"879894ab-6a37-4e4e-8427-2d199eaaa51e","Type":"ContainerStarted","Data":"5732e6afd4edf2b6fae21dafd669d43dc94930af7eb95cf43fa61c7647991794"} Jul 21 02:10:44.059405 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:44.059338 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-24k47" podStartSLOduration=39.047228383 podStartE2EDuration="41.059316354s" totalImagesPullingTime="2.012087971s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 02:10:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:10:44.058883287 +0000 UTC m=+78.078377211" watchObservedRunningTime="2026-07-21 02:10:44.059316354 +0000 UTC m=+78.078810238" Jul 21 02:10:44.076514 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:44.076462 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-7hdf7" podStartSLOduration=2.859282142 podStartE2EDuration="4.076442306s" totalImagesPullingTime="1.217160164s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 02:10:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:10:44.075719925 +0000 UTC m=+78.095213839" watchObservedRunningTime="2026-07-21 02:10:44.076442306 +0000 UTC m=+78.095936225" Jul 21 02:10:45.042723 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:45.042689 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-7hdf7" Jul 21 02:10:45.047504 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:45.047477 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-7hdf7" Jul 21 02:10:45.779869 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:45.779790 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-6g4s2"] Jul 21 02:10:45.782837 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:45.782820 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-6g4s2" Jul 21 02:10:45.787077 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:45.787055 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 21 02:10:45.787385 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:45.787362 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 21 02:10:45.787631 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:45.787407 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 21 02:10:45.787631 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:45.787430 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 21 02:10:45.787631 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:45.787489 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-2s7gp\"" Jul 21 02:10:45.787821 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:45.787668 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 21 02:10:45.791250 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:45.791230 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-6g4s2"] Jul 21 02:10:45.897490 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:45.897449 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7scxl\" (UniqueName: \"kubernetes.io/projected/ea35bdb8-4979-46f2-b79f-00ca18560d1a-kube-api-access-7scxl\") pod \"prometheus-operator-5945b5cfd-6g4s2\" (UID: \"ea35bdb8-4979-46f2-b79f-00ca18560d1a\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-6g4s2" Jul 21 02:10:45.897659 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:45.897503 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea35bdb8-4979-46f2-b79f-00ca18560d1a-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-6g4s2\" (UID: \"ea35bdb8-4979-46f2-b79f-00ca18560d1a\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-6g4s2" Jul 21 02:10:45.897659 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:45.897523 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ea35bdb8-4979-46f2-b79f-00ca18560d1a-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-6g4s2\" (UID: \"ea35bdb8-4979-46f2-b79f-00ca18560d1a\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-6g4s2" Jul 21 02:10:45.897659 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:45.897576 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ea35bdb8-4979-46f2-b79f-00ca18560d1a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-6g4s2\" (UID: \"ea35bdb8-4979-46f2-b79f-00ca18560d1a\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-6g4s2" Jul 21 02:10:45.998541 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:45.998509 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7scxl\" (UniqueName: \"kubernetes.io/projected/ea35bdb8-4979-46f2-b79f-00ca18560d1a-kube-api-access-7scxl\") pod \"prometheus-operator-5945b5cfd-6g4s2\" (UID: \"ea35bdb8-4979-46f2-b79f-00ca18560d1a\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-6g4s2" Jul 21 02:10:45.998680 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:45.998557 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea35bdb8-4979-46f2-b79f-00ca18560d1a-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-6g4s2\" (UID: \"ea35bdb8-4979-46f2-b79f-00ca18560d1a\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-6g4s2" Jul 21 02:10:45.998680 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:45.998573 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ea35bdb8-4979-46f2-b79f-00ca18560d1a-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-6g4s2\" (UID: \"ea35bdb8-4979-46f2-b79f-00ca18560d1a\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-6g4s2" Jul 21 02:10:45.998826 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:45.998805 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ea35bdb8-4979-46f2-b79f-00ca18560d1a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-6g4s2\" (UID: \"ea35bdb8-4979-46f2-b79f-00ca18560d1a\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-6g4s2" Jul 21 02:10:45.999212 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:45.999193 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea35bdb8-4979-46f2-b79f-00ca18560d1a-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-6g4s2\" (UID: \"ea35bdb8-4979-46f2-b79f-00ca18560d1a\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-6g4s2" Jul 21 02:10:46.000996 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:46.000975 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ea35bdb8-4979-46f2-b79f-00ca18560d1a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-6g4s2\" (UID: \"ea35bdb8-4979-46f2-b79f-00ca18560d1a\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-6g4s2" Jul 21 02:10:46.001062 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:46.000994 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ea35bdb8-4979-46f2-b79f-00ca18560d1a-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-6g4s2\" (UID: \"ea35bdb8-4979-46f2-b79f-00ca18560d1a\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-6g4s2" Jul 21 02:10:46.007099 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:46.007077 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7scxl\" (UniqueName: \"kubernetes.io/projected/ea35bdb8-4979-46f2-b79f-00ca18560d1a-kube-api-access-7scxl\") pod \"prometheus-operator-5945b5cfd-6g4s2\" (UID: \"ea35bdb8-4979-46f2-b79f-00ca18560d1a\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-6g4s2" Jul 21 02:10:46.047507 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:46.047445 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ncxvc" event={"ID":"a89ab6c0-0bd2-4cb5-aca9-71b07c462005","Type":"ContainerStarted","Data":"1069e0253df7280d901c8aa7c3f8488af26219d0712920813d2b0d068eadeec6"} Jul 21 02:10:46.072696 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:46.072652 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-ncxvc" podStartSLOduration=17.551722132 podStartE2EDuration="35.072637557s" totalImagesPullingTime="17.520915425s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 02:10:11 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:10:46.071667827 +0000 UTC m=+80.091161750" watchObservedRunningTime="2026-07-21 02:10:46.072637557 +0000 UTC m=+80.092131461" Jul 21 02:10:46.092191 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:46.092171 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-6g4s2" Jul 21 02:10:46.210867 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:46.210740 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-6g4s2"] Jul 21 02:10:46.213357 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:10:46.213330 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea35bdb8_4979_46f2_b79f_00ca18560d1a.slice/crio-dd292d11202e66f168b11f9d3e6da5bdeb2ebfb21dc728609a157c921c172ef6 WatchSource:0}: Error finding container dd292d11202e66f168b11f9d3e6da5bdeb2ebfb21dc728609a157c921c172ef6: Status 404 returned error can't find the container with id dd292d11202e66f168b11f9d3e6da5bdeb2ebfb21dc728609a157c921c172ef6 Jul 21 02:10:47.052990 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:47.052948 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-6g4s2" event={"ID":"ea35bdb8-4979-46f2-b79f-00ca18560d1a","Type":"ContainerStarted","Data":"dd292d11202e66f168b11f9d3e6da5bdeb2ebfb21dc728609a157c921c172ef6"} Jul 21 02:10:48.060555 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:48.060515 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-6g4s2" event={"ID":"ea35bdb8-4979-46f2-b79f-00ca18560d1a","Type":"ContainerStarted","Data":"ab89688f3a55904c055d4628f17a6da361be0d46dc53b201d3b74b525168483f"} Jul 21 02:10:48.060939 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:48.060567 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-6g4s2" event={"ID":"ea35bdb8-4979-46f2-b79f-00ca18560d1a","Type":"ContainerStarted","Data":"e43ebb41a309bc164726c1de4bd1ffc8e35a0da7e8441c79713d2da4f5ef8fd6"} Jul 21 02:10:48.080093 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:48.080040 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5945b5cfd-6g4s2" podStartSLOduration=2.037750458 podStartE2EDuration="3.080025619s" totalImagesPullingTime="1.042275161s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 02:10:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:10:48.078787772 +0000 UTC m=+82.098281678" watchObservedRunningTime="2026-07-21 02:10:48.080025619 +0000 UTC m=+82.099519522" Jul 21 02:10:50.236069 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.236024 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-7llbd"] Jul 21 02:10:50.239437 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.239410 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-7llbd" Jul 21 02:10:50.242390 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.242371 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-ggxts\"" Jul 21 02:10:50.242505 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.242373 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 21 02:10:50.242505 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.242373 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 21 02:10:50.253656 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.253633 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-7llbd"] Jul 21 02:10:50.281646 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.281618 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-kctq6"] Jul 21 02:10:50.284565 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.284549 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-kctq6" Jul 21 02:10:50.287625 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.287607 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 21 02:10:50.288099 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.288081 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-jrbvv\"" Jul 21 02:10:50.288208 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.288098 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 21 02:10:50.288208 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.288122 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 21 02:10:50.301161 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.301138 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-6rb7p"] Jul 21 02:10:50.304134 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.304115 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6rb7p" Jul 21 02:10:50.306660 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.306637 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-kctq6"] Jul 21 02:10:50.306796 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.306780 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 21 02:10:50.306865 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.306804 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-bhkmf\"" Jul 21 02:10:50.307131 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.307116 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 21 02:10:50.307215 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.307138 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 21 02:10:50.332367 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.332333 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fndk4\" (UniqueName: \"kubernetes.io/projected/0cd36c63-e03e-4496-87ae-ffaecb2434c4-kube-api-access-fndk4\") pod \"openshift-state-metrics-79f6958cdf-7llbd\" (UID: \"0cd36c63-e03e-4496-87ae-ffaecb2434c4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-7llbd" Jul 21 02:10:50.332367 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.332369 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b393906b-0a2d-4032-b8c4-0283c219d4c9-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-kctq6\" (UID: \"b393906b-0a2d-4032-b8c4-0283c219d4c9\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-kctq6" Jul 21 02:10:50.332555 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.332390 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b393906b-0a2d-4032-b8c4-0283c219d4c9-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-kctq6\" (UID: \"b393906b-0a2d-4032-b8c4-0283c219d4c9\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-kctq6" Jul 21 02:10:50.332555 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.332432 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0cd36c63-e03e-4496-87ae-ffaecb2434c4-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-7llbd\" (UID: \"0cd36c63-e03e-4496-87ae-ffaecb2434c4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-7llbd" Jul 21 02:10:50.332555 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.332457 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0cd36c63-e03e-4496-87ae-ffaecb2434c4-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-7llbd\" (UID: \"0cd36c63-e03e-4496-87ae-ffaecb2434c4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-7llbd" Jul 21 02:10:50.332555 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.332484 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b393906b-0a2d-4032-b8c4-0283c219d4c9-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-kctq6\" (UID: \"b393906b-0a2d-4032-b8c4-0283c219d4c9\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-kctq6" Jul 21 02:10:50.332555 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.332521 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b393906b-0a2d-4032-b8c4-0283c219d4c9-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-kctq6\" (UID: \"b393906b-0a2d-4032-b8c4-0283c219d4c9\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-kctq6" Jul 21 02:10:50.332702 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.332561 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0cd36c63-e03e-4496-87ae-ffaecb2434c4-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-7llbd\" (UID: \"0cd36c63-e03e-4496-87ae-ffaecb2434c4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-7llbd" Jul 21 02:10:50.332702 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.332602 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b7gp\" (UniqueName: \"kubernetes.io/projected/b393906b-0a2d-4032-b8c4-0283c219d4c9-kube-api-access-7b7gp\") pod \"kube-state-metrics-56c5594674-kctq6\" (UID: \"b393906b-0a2d-4032-b8c4-0283c219d4c9\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-kctq6" Jul 21 02:10:50.332702 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.332633 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b393906b-0a2d-4032-b8c4-0283c219d4c9-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-kctq6\" (UID: \"b393906b-0a2d-4032-b8c4-0283c219d4c9\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-kctq6" Jul 21 02:10:50.433348 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.433318 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fndk4\" (UniqueName: \"kubernetes.io/projected/0cd36c63-e03e-4496-87ae-ffaecb2434c4-kube-api-access-fndk4\") pod \"openshift-state-metrics-79f6958cdf-7llbd\" (UID: \"0cd36c63-e03e-4496-87ae-ffaecb2434c4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-7llbd" Jul 21 02:10:50.433348 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.433350 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b393906b-0a2d-4032-b8c4-0283c219d4c9-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-kctq6\" (UID: \"b393906b-0a2d-4032-b8c4-0283c219d4c9\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-kctq6" Jul 21 02:10:50.433591 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.433368 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b393906b-0a2d-4032-b8c4-0283c219d4c9-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-kctq6\" (UID: \"b393906b-0a2d-4032-b8c4-0283c219d4c9\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-kctq6" Jul 21 02:10:50.433591 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.433552 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/778daaa9-d1f9-4ee5-8f24-8c291a24f197-node-exporter-wtmp\") pod \"node-exporter-6rb7p\" (UID: \"778daaa9-d1f9-4ee5-8f24-8c291a24f197\") " pod="openshift-monitoring/node-exporter-6rb7p" Jul 21 02:10:50.433705 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.433598 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0cd36c63-e03e-4496-87ae-ffaecb2434c4-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-7llbd\" (UID: \"0cd36c63-e03e-4496-87ae-ffaecb2434c4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-7llbd" Jul 21 02:10:50.433705 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.433627 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/778daaa9-d1f9-4ee5-8f24-8c291a24f197-sys\") pod \"node-exporter-6rb7p\" (UID: \"778daaa9-d1f9-4ee5-8f24-8c291a24f197\") " pod="openshift-monitoring/node-exporter-6rb7p" Jul 21 02:10:50.433705 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.433652 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/778daaa9-d1f9-4ee5-8f24-8c291a24f197-node-exporter-accelerators-collector-config\") pod \"node-exporter-6rb7p\" (UID: \"778daaa9-d1f9-4ee5-8f24-8c291a24f197\") " pod="openshift-monitoring/node-exporter-6rb7p" Jul 21 02:10:50.433705 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.433698 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0cd36c63-e03e-4496-87ae-ffaecb2434c4-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-7llbd\" (UID: \"0cd36c63-e03e-4496-87ae-ffaecb2434c4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-7llbd" Jul 21 02:10:50.433865 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.433763 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvq5z\" (UniqueName: \"kubernetes.io/projected/778daaa9-d1f9-4ee5-8f24-8c291a24f197-kube-api-access-bvq5z\") pod \"node-exporter-6rb7p\" (UID: \"778daaa9-d1f9-4ee5-8f24-8c291a24f197\") " pod="openshift-monitoring/node-exporter-6rb7p" Jul 21 02:10:50.433865 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.433775 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b393906b-0a2d-4032-b8c4-0283c219d4c9-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-kctq6\" (UID: \"b393906b-0a2d-4032-b8c4-0283c219d4c9\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-kctq6" Jul 21 02:10:50.433865 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.433800 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/778daaa9-d1f9-4ee5-8f24-8c291a24f197-node-exporter-textfile\") pod \"node-exporter-6rb7p\" (UID: \"778daaa9-d1f9-4ee5-8f24-8c291a24f197\") " pod="openshift-monitoring/node-exporter-6rb7p" Jul 21 02:10:50.433865 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.433846 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b393906b-0a2d-4032-b8c4-0283c219d4c9-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-kctq6\" (UID: \"b393906b-0a2d-4032-b8c4-0283c219d4c9\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-kctq6" Jul 21 02:10:50.433865 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.433865 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/778daaa9-d1f9-4ee5-8f24-8c291a24f197-root\") pod \"node-exporter-6rb7p\" (UID: \"778daaa9-d1f9-4ee5-8f24-8c291a24f197\") " pod="openshift-monitoring/node-exporter-6rb7p" Jul 21 02:10:50.434137 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.433884 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b393906b-0a2d-4032-b8c4-0283c219d4c9-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-kctq6\" (UID: \"b393906b-0a2d-4032-b8c4-0283c219d4c9\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-kctq6" Jul 21 02:10:50.434137 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.433920 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0cd36c63-e03e-4496-87ae-ffaecb2434c4-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-7llbd\" (UID: \"0cd36c63-e03e-4496-87ae-ffaecb2434c4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-7llbd" Jul 21 02:10:50.434137 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.434029 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b7gp\" (UniqueName: \"kubernetes.io/projected/b393906b-0a2d-4032-b8c4-0283c219d4c9-kube-api-access-7b7gp\") pod \"kube-state-metrics-56c5594674-kctq6\" (UID: \"b393906b-0a2d-4032-b8c4-0283c219d4c9\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-kctq6" Jul 21 02:10:50.434137 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.434088 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/778daaa9-d1f9-4ee5-8f24-8c291a24f197-metrics-client-ca\") pod \"node-exporter-6rb7p\" (UID: \"778daaa9-d1f9-4ee5-8f24-8c291a24f197\") " pod="openshift-monitoring/node-exporter-6rb7p" Jul 21 02:10:50.434137 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.434107 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b393906b-0a2d-4032-b8c4-0283c219d4c9-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-kctq6\" (UID: \"b393906b-0a2d-4032-b8c4-0283c219d4c9\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-kctq6" Jul 21 02:10:50.434137 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.434118 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b393906b-0a2d-4032-b8c4-0283c219d4c9-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-kctq6\" (UID: \"b393906b-0a2d-4032-b8c4-0283c219d4c9\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-kctq6" Jul 21 02:10:50.434462 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.434148 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/778daaa9-d1f9-4ee5-8f24-8c291a24f197-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6rb7p\" (UID: \"778daaa9-d1f9-4ee5-8f24-8c291a24f197\") " pod="openshift-monitoring/node-exporter-6rb7p" Jul 21 02:10:50.434462 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.434181 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/778daaa9-d1f9-4ee5-8f24-8c291a24f197-node-exporter-tls\") pod \"node-exporter-6rb7p\" (UID: \"778daaa9-d1f9-4ee5-8f24-8c291a24f197\") " pod="openshift-monitoring/node-exporter-6rb7p" Jul 21 02:10:50.434462 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.434448 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0cd36c63-e03e-4496-87ae-ffaecb2434c4-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-7llbd\" (UID: \"0cd36c63-e03e-4496-87ae-ffaecb2434c4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-7llbd" Jul 21 02:10:50.434619 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.434528 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b393906b-0a2d-4032-b8c4-0283c219d4c9-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-kctq6\" (UID: \"b393906b-0a2d-4032-b8c4-0283c219d4c9\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-kctq6" Jul 21 02:10:50.436407 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.436384 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0cd36c63-e03e-4496-87ae-ffaecb2434c4-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-7llbd\" (UID: \"0cd36c63-e03e-4496-87ae-ffaecb2434c4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-7llbd" Jul 21 02:10:50.436407 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.436395 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b393906b-0a2d-4032-b8c4-0283c219d4c9-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-kctq6\" (UID: \"b393906b-0a2d-4032-b8c4-0283c219d4c9\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-kctq6" Jul 21 02:10:50.436539 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.436452 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b393906b-0a2d-4032-b8c4-0283c219d4c9-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-kctq6\" (UID: \"b393906b-0a2d-4032-b8c4-0283c219d4c9\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-kctq6" Jul 21 02:10:50.436539 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.436484 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0cd36c63-e03e-4496-87ae-ffaecb2434c4-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-7llbd\" (UID: \"0cd36c63-e03e-4496-87ae-ffaecb2434c4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-7llbd" Jul 21 02:10:50.443046 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.443022 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b7gp\" (UniqueName: \"kubernetes.io/projected/b393906b-0a2d-4032-b8c4-0283c219d4c9-kube-api-access-7b7gp\") pod \"kube-state-metrics-56c5594674-kctq6\" (UID: \"b393906b-0a2d-4032-b8c4-0283c219d4c9\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-kctq6" Jul 21 02:10:50.443139 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.443021 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fndk4\" (UniqueName: \"kubernetes.io/projected/0cd36c63-e03e-4496-87ae-ffaecb2434c4-kube-api-access-fndk4\") pod \"openshift-state-metrics-79f6958cdf-7llbd\" (UID: \"0cd36c63-e03e-4496-87ae-ffaecb2434c4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-7llbd" Jul 21 02:10:50.534598 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.534520 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/778daaa9-d1f9-4ee5-8f24-8c291a24f197-metrics-client-ca\") pod \"node-exporter-6rb7p\" (UID: \"778daaa9-d1f9-4ee5-8f24-8c291a24f197\") " pod="openshift-monitoring/node-exporter-6rb7p" Jul 21 02:10:50.534598 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.534553 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/778daaa9-d1f9-4ee5-8f24-8c291a24f197-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6rb7p\" (UID: \"778daaa9-d1f9-4ee5-8f24-8c291a24f197\") " pod="openshift-monitoring/node-exporter-6rb7p" Jul 21 02:10:50.534598 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.534570 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/778daaa9-d1f9-4ee5-8f24-8c291a24f197-node-exporter-tls\") pod \"node-exporter-6rb7p\" (UID: \"778daaa9-d1f9-4ee5-8f24-8c291a24f197\") " pod="openshift-monitoring/node-exporter-6rb7p" Jul 21 02:10:50.534598 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.534595 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/778daaa9-d1f9-4ee5-8f24-8c291a24f197-node-exporter-wtmp\") pod \"node-exporter-6rb7p\" (UID: \"778daaa9-d1f9-4ee5-8f24-8c291a24f197\") " pod="openshift-monitoring/node-exporter-6rb7p" Jul 21 02:10:50.534895 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.534612 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/778daaa9-d1f9-4ee5-8f24-8c291a24f197-sys\") pod \"node-exporter-6rb7p\" (UID: \"778daaa9-d1f9-4ee5-8f24-8c291a24f197\") " pod="openshift-monitoring/node-exporter-6rb7p" Jul 21 02:10:50.534895 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.534653 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/778daaa9-d1f9-4ee5-8f24-8c291a24f197-sys\") pod \"node-exporter-6rb7p\" (UID: \"778daaa9-d1f9-4ee5-8f24-8c291a24f197\") " pod="openshift-monitoring/node-exporter-6rb7p" Jul 21 02:10:50.534895 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.534650 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/778daaa9-d1f9-4ee5-8f24-8c291a24f197-node-exporter-accelerators-collector-config\") pod \"node-exporter-6rb7p\" (UID: \"778daaa9-d1f9-4ee5-8f24-8c291a24f197\") " pod="openshift-monitoring/node-exporter-6rb7p" Jul 21 02:10:50.534895 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.534711 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvq5z\" (UniqueName: \"kubernetes.io/projected/778daaa9-d1f9-4ee5-8f24-8c291a24f197-kube-api-access-bvq5z\") pod \"node-exporter-6rb7p\" (UID: \"778daaa9-d1f9-4ee5-8f24-8c291a24f197\") " pod="openshift-monitoring/node-exporter-6rb7p" Jul 21 02:10:50.534895 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.534738 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/778daaa9-d1f9-4ee5-8f24-8c291a24f197-node-exporter-textfile\") pod \"node-exporter-6rb7p\" (UID: \"778daaa9-d1f9-4ee5-8f24-8c291a24f197\") " pod="openshift-monitoring/node-exporter-6rb7p" Jul 21 02:10:50.534895 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.534766 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/778daaa9-d1f9-4ee5-8f24-8c291a24f197-root\") pod \"node-exporter-6rb7p\" (UID: \"778daaa9-d1f9-4ee5-8f24-8c291a24f197\") " pod="openshift-monitoring/node-exporter-6rb7p" Jul 21 02:10:50.534895 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.534779 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/778daaa9-d1f9-4ee5-8f24-8c291a24f197-node-exporter-wtmp\") pod \"node-exporter-6rb7p\" (UID: \"778daaa9-d1f9-4ee5-8f24-8c291a24f197\") " pod="openshift-monitoring/node-exporter-6rb7p" Jul 21 02:10:50.534895 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.534853 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/778daaa9-d1f9-4ee5-8f24-8c291a24f197-root\") pod \"node-exporter-6rb7p\" (UID: \"778daaa9-d1f9-4ee5-8f24-8c291a24f197\") " pod="openshift-monitoring/node-exporter-6rb7p" Jul 21 02:10:50.535184 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.535068 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/778daaa9-d1f9-4ee5-8f24-8c291a24f197-node-exporter-textfile\") pod \"node-exporter-6rb7p\" (UID: \"778daaa9-d1f9-4ee5-8f24-8c291a24f197\") " pod="openshift-monitoring/node-exporter-6rb7p" Jul 21 02:10:50.535184 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.535151 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/778daaa9-d1f9-4ee5-8f24-8c291a24f197-node-exporter-accelerators-collector-config\") pod \"node-exporter-6rb7p\" (UID: \"778daaa9-d1f9-4ee5-8f24-8c291a24f197\") " pod="openshift-monitoring/node-exporter-6rb7p" Jul 21 02:10:50.535297 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.535221 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/778daaa9-d1f9-4ee5-8f24-8c291a24f197-metrics-client-ca\") pod \"node-exporter-6rb7p\" (UID: \"778daaa9-d1f9-4ee5-8f24-8c291a24f197\") " pod="openshift-monitoring/node-exporter-6rb7p" Jul 21 02:10:50.536943 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.536920 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/778daaa9-d1f9-4ee5-8f24-8c291a24f197-node-exporter-tls\") pod \"node-exporter-6rb7p\" (UID: \"778daaa9-d1f9-4ee5-8f24-8c291a24f197\") " pod="openshift-monitoring/node-exporter-6rb7p" Jul 21 02:10:50.537039 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.536966 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/778daaa9-d1f9-4ee5-8f24-8c291a24f197-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6rb7p\" (UID: \"778daaa9-d1f9-4ee5-8f24-8c291a24f197\") " pod="openshift-monitoring/node-exporter-6rb7p" Jul 21 02:10:50.544235 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.544218 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvq5z\" (UniqueName: \"kubernetes.io/projected/778daaa9-d1f9-4ee5-8f24-8c291a24f197-kube-api-access-bvq5z\") pod \"node-exporter-6rb7p\" (UID: \"778daaa9-d1f9-4ee5-8f24-8c291a24f197\") " pod="openshift-monitoring/node-exporter-6rb7p" Jul 21 02:10:50.548152 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.548134 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-7llbd" Jul 21 02:10:50.594468 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.594096 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-kctq6" Jul 21 02:10:50.613896 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.613680 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6rb7p" Jul 21 02:10:50.625493 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:10:50.625466 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod778daaa9_d1f9_4ee5_8f24_8c291a24f197.slice/crio-239ed318c642d9508871979f699fad1ab145e4e4bb04c76284f24ee84bf828a3 WatchSource:0}: Error finding container 239ed318c642d9508871979f699fad1ab145e4e4bb04c76284f24ee84bf828a3: Status 404 returned error can't find the container with id 239ed318c642d9508871979f699fad1ab145e4e4bb04c76284f24ee84bf828a3 Jul 21 02:10:50.702017 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.701645 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-7llbd"] Jul 21 02:10:50.703907 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:10:50.703880 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cd36c63_e03e_4496_87ae_ffaecb2434c4.slice/crio-75cc8cb02957472e3cf07c55416c2ef6d8c4803dfc43385b9184e33488d925d0 WatchSource:0}: Error finding container 75cc8cb02957472e3cf07c55416c2ef6d8c4803dfc43385b9184e33488d925d0: Status 404 returned error can't find the container with id 75cc8cb02957472e3cf07c55416c2ef6d8c4803dfc43385b9184e33488d925d0 Jul 21 02:10:50.734428 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:50.734400 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-kctq6"] Jul 21 02:10:50.738420 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:10:50.738397 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb393906b_0a2d_4032_b8c4_0283c219d4c9.slice/crio-42a04b1e100859d55eaaadcea3735158b34705c81c5af403f864c19eade693eb WatchSource:0}: Error finding container 42a04b1e100859d55eaaadcea3735158b34705c81c5af403f864c19eade693eb: Status 404 returned error can't find the container with id 42a04b1e100859d55eaaadcea3735158b34705c81c5af403f864c19eade693eb Jul 21 02:10:51.072393 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:51.072301 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6rb7p" event={"ID":"778daaa9-d1f9-4ee5-8f24-8c291a24f197","Type":"ContainerStarted","Data":"239ed318c642d9508871979f699fad1ab145e4e4bb04c76284f24ee84bf828a3"} Jul 21 02:10:51.073663 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:51.073631 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-kctq6" event={"ID":"b393906b-0a2d-4032-b8c4-0283c219d4c9","Type":"ContainerStarted","Data":"42a04b1e100859d55eaaadcea3735158b34705c81c5af403f864c19eade693eb"} Jul 21 02:10:51.075491 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:51.075465 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-7llbd" event={"ID":"0cd36c63-e03e-4496-87ae-ffaecb2434c4","Type":"ContainerStarted","Data":"e707d48905b3eccd7287d855dc8c75070e4f9d53dc78a933dba4cc31fb88de3b"} Jul 21 02:10:51.075613 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:51.075521 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-7llbd" event={"ID":"0cd36c63-e03e-4496-87ae-ffaecb2434c4","Type":"ContainerStarted","Data":"52d1fd736bd26408256a6082302fe8afd2e0496b0d720d3b89adbed9c16ebd48"} Jul 21 02:10:51.075613 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:51.075549 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-7llbd" event={"ID":"0cd36c63-e03e-4496-87ae-ffaecb2434c4","Type":"ContainerStarted","Data":"75cc8cb02957472e3cf07c55416c2ef6d8c4803dfc43385b9184e33488d925d0"} Jul 21 02:10:51.279003 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:51.278970 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 21 02:10:51.283404 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:51.283378 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 21 02:10:51.286060 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:51.286036 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 21 02:10:51.286170 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:51.286080 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 21 02:10:51.286170 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:51.286155 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 21 02:10:51.286308 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:51.286160 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 21 02:10:51.286456 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:51.286439 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 21 02:10:51.286517 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:51.286439 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 21 02:10:51.286517 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:51.286488 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 21 02:10:51.286603 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:51.286583 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-xhfj8\"" Jul 21 02:10:51.286918 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:51.286902 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 21 02:10:51.286918 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:51.286911 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 21 02:10:51.300396 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:51.300371 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 21 02:10:51.341308 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:51.341200 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn9kj\" (UniqueName: \"kubernetes.io/projected/2694a3ee-0897-4753-bfee-ff58550df666-kube-api-access-mn9kj\") pod \"alertmanager-main-0\" (UID: \"2694a3ee-0897-4753-bfee-ff58550df666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 02:10:51.341308 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:51.341251 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2694a3ee-0897-4753-bfee-ff58550df666-config-volume\") pod \"alertmanager-main-0\" (UID: \"2694a3ee-0897-4753-bfee-ff58550df666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 02:10:51.341308 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:51.341301 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2694a3ee-0897-4753-bfee-ff58550df666-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2694a3ee-0897-4753-bfee-ff58550df666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 02:10:51.341555 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:51.341324 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2694a3ee-0897-4753-bfee-ff58550df666-web-config\") pod \"alertmanager-main-0\" (UID: \"2694a3ee-0897-4753-bfee-ff58550df666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 02:10:51.341555 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:51.341391 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2694a3ee-0897-4753-bfee-ff58550df666-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2694a3ee-0897-4753-bfee-ff58550df666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 02:10:51.341555 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:51.341425 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2694a3ee-0897-4753-bfee-ff58550df666-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2694a3ee-0897-4753-bfee-ff58550df666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 02:10:51.341555 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:51.341463 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2694a3ee-0897-4753-bfee-ff58550df666-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2694a3ee-0897-4753-bfee-ff58550df666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 02:10:51.341555 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:51.341510 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2694a3ee-0897-4753-bfee-ff58550df666-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2694a3ee-0897-4753-bfee-ff58550df666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 02:10:51.341555 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:51.341535 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2694a3ee-0897-4753-bfee-ff58550df666-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2694a3ee-0897-4753-bfee-ff58550df666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 02:10:51.341804 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:51.341578 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2694a3ee-0897-4753-bfee-ff58550df666-config-out\") pod \"alertmanager-main-0\" (UID: \"2694a3ee-0897-4753-bfee-ff58550df666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 02:10:51.342098 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:51.342072 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2694a3ee-0897-4753-bfee-ff58550df666-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2694a3ee-0897-4753-bfee-ff58550df666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 02:10:51.342211 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:51.342121 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2694a3ee-0897-4753-bfee-ff58550df666-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2694a3ee-0897-4753-bfee-ff58550df666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 02:10:51.342211 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:51.342150 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2694a3ee-0897-4753-bfee-ff58550df666-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2694a3ee-0897-4753-bfee-ff58550df666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 02:10:51.449883 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:51.449842 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2694a3ee-0897-4753-bfee-ff58550df666-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2694a3ee-0897-4753-bfee-ff58550df666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 02:10:51.450055 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:51.449919 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2694a3ee-0897-4753-bfee-ff58550df666-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2694a3ee-0897-4753-bfee-ff58550df666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 02:10:51.450055 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:51.449952 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2694a3ee-0897-4753-bfee-ff58550df666-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2694a3ee-0897-4753-bfee-ff58550df666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 02:10:51.450055 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:51.449980 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn9kj\" (UniqueName: \"kubernetes.io/projected/2694a3ee-0897-4753-bfee-ff58550df666-kube-api-access-mn9kj\") pod \"alertmanager-main-0\" (UID: \"2694a3ee-0897-4753-bfee-ff58550df666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 02:10:51.450055 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:51.450022 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2694a3ee-0897-4753-bfee-ff58550df666-config-volume\") pod \"alertmanager-main-0\" (UID: \"2694a3ee-0897-4753-bfee-ff58550df666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 02:10:51.450055 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:10:51.450046 2574 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 21 02:10:51.450289 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:51.450058 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2694a3ee-0897-4753-bfee-ff58550df666-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2694a3ee-0897-4753-bfee-ff58550df666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 02:10:51.450289 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:51.450081 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2694a3ee-0897-4753-bfee-ff58550df666-web-config\") pod \"alertmanager-main-0\" (UID: \"2694a3ee-0897-4753-bfee-ff58550df666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 02:10:51.450289 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:10:51.450123 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2694a3ee-0897-4753-bfee-ff58550df666-secret-alertmanager-main-tls podName:2694a3ee-0897-4753-bfee-ff58550df666 nodeName:}" failed. No retries permitted until 2026-07-21 02:10:51.950099547 +0000 UTC m=+85.969593432 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/2694a3ee-0897-4753-bfee-ff58550df666-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "2694a3ee-0897-4753-bfee-ff58550df666") : secret "alertmanager-main-tls" not found Jul 21 02:10:51.450289 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:51.450167 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2694a3ee-0897-4753-bfee-ff58550df666-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2694a3ee-0897-4753-bfee-ff58550df666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 02:10:51.450289 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:51.450194 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2694a3ee-0897-4753-bfee-ff58550df666-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2694a3ee-0897-4753-bfee-ff58550df666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 02:10:51.450289 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:51.450232 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2694a3ee-0897-4753-bfee-ff58550df666-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2694a3ee-0897-4753-bfee-ff58550df666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 02:10:51.450532 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:51.450291 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2694a3ee-0897-4753-bfee-ff58550df666-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2694a3ee-0897-4753-bfee-ff58550df666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 02:10:51.450532 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:51.450324 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2694a3ee-0897-4753-bfee-ff58550df666-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2694a3ee-0897-4753-bfee-ff58550df666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 02:10:51.450532 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:51.450349 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2694a3ee-0897-4753-bfee-ff58550df666-config-out\") pod \"alertmanager-main-0\" (UID: \"2694a3ee-0897-4753-bfee-ff58550df666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 02:10:51.450532 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:10:51.450515 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2694a3ee-0897-4753-bfee-ff58550df666-alertmanager-trusted-ca-bundle podName:2694a3ee-0897-4753-bfee-ff58550df666 nodeName:}" failed. No retries permitted until 2026-07-21 02:10:51.950492113 +0000 UTC m=+85.969986015 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/2694a3ee-0897-4753-bfee-ff58550df666-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "2694a3ee-0897-4753-bfee-ff58550df666") : configmap references non-existent config key: ca-bundle.crt Jul 21 02:10:51.450695 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:51.450676 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2694a3ee-0897-4753-bfee-ff58550df666-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2694a3ee-0897-4753-bfee-ff58550df666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 02:10:51.451986 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:51.451638 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2694a3ee-0897-4753-bfee-ff58550df666-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2694a3ee-0897-4753-bfee-ff58550df666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 02:10:51.455146 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:51.455083 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2694a3ee-0897-4753-bfee-ff58550df666-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2694a3ee-0897-4753-bfee-ff58550df666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 02:10:51.456209 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:51.456155 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2694a3ee-0897-4753-bfee-ff58550df666-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2694a3ee-0897-4753-bfee-ff58550df666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 02:10:51.456321 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:51.456303 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2694a3ee-0897-4753-bfee-ff58550df666-web-config\") pod \"alertmanager-main-0\" (UID: \"2694a3ee-0897-4753-bfee-ff58550df666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 02:10:51.457041 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:51.457000 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2694a3ee-0897-4753-bfee-ff58550df666-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2694a3ee-0897-4753-bfee-ff58550df666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 02:10:51.457150 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:51.457096 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2694a3ee-0897-4753-bfee-ff58550df666-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2694a3ee-0897-4753-bfee-ff58550df666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 02:10:51.457722 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:51.457677 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2694a3ee-0897-4753-bfee-ff58550df666-config-volume\") pod \"alertmanager-main-0\" (UID: \"2694a3ee-0897-4753-bfee-ff58550df666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 02:10:51.458329 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:51.458289 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2694a3ee-0897-4753-bfee-ff58550df666-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2694a3ee-0897-4753-bfee-ff58550df666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 02:10:51.458440 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:51.458386 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2694a3ee-0897-4753-bfee-ff58550df666-config-out\") pod \"alertmanager-main-0\" (UID: \"2694a3ee-0897-4753-bfee-ff58550df666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 02:10:51.475206 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:51.475181 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn9kj\" (UniqueName: \"kubernetes.io/projected/2694a3ee-0897-4753-bfee-ff58550df666-kube-api-access-mn9kj\") pod \"alertmanager-main-0\" (UID: \"2694a3ee-0897-4753-bfee-ff58550df666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 02:10:51.956169 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:51.956138 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2694a3ee-0897-4753-bfee-ff58550df666-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2694a3ee-0897-4753-bfee-ff58550df666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 02:10:51.956348 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:51.956193 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2694a3ee-0897-4753-bfee-ff58550df666-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2694a3ee-0897-4753-bfee-ff58550df666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 02:10:51.957058 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:51.957031 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2694a3ee-0897-4753-bfee-ff58550df666-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2694a3ee-0897-4753-bfee-ff58550df666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 02:10:51.958789 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:51.958763 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2694a3ee-0897-4753-bfee-ff58550df666-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2694a3ee-0897-4753-bfee-ff58550df666\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 02:10:52.081111 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:52.081072 2574 generic.go:356] "Generic (PLEG): container finished" podID="778daaa9-d1f9-4ee5-8f24-8c291a24f197" containerID="7e3b68c11909b2a200ddd8b41c5178fa00eb2ef35789c9d0e0bdd1fa073ae8d7" exitCode=0 Jul 21 02:10:52.081318 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:52.081170 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6rb7p" event={"ID":"778daaa9-d1f9-4ee5-8f24-8c291a24f197","Type":"ContainerDied","Data":"7e3b68c11909b2a200ddd8b41c5178fa00eb2ef35789c9d0e0bdd1fa073ae8d7"} Jul 21 02:10:52.193491 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:52.193455 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 21 02:10:52.448609 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:52.448584 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 21 02:10:53.086891 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:53.086858 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2694a3ee-0897-4753-bfee-ff58550df666","Type":"ContainerStarted","Data":"3f704f290a811d21b14b4e999ccaec48bbeb5209d866aedb8e73f62d6a0fd521"} Jul 21 02:10:53.089400 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:53.089368 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6rb7p" event={"ID":"778daaa9-d1f9-4ee5-8f24-8c291a24f197","Type":"ContainerStarted","Data":"777ac726816fd18aec7cc7787fa5d080e09368c5895b4337fd0e263401d10d29"} Jul 21 02:10:53.089542 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:53.089422 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6rb7p" event={"ID":"778daaa9-d1f9-4ee5-8f24-8c291a24f197","Type":"ContainerStarted","Data":"7cb9670e1a6040f04e8cd124e7a0dcd4a84031e4d0e0ae2c85128bcdf04f848e"} Jul 21 02:10:53.091670 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:53.091641 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-kctq6" event={"ID":"b393906b-0a2d-4032-b8c4-0283c219d4c9","Type":"ContainerStarted","Data":"160c6b0d544d78bf961d8ace4886aa3b38e279ee323e15c1834483f9093ce126"} Jul 21 02:10:53.091784 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:53.091703 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-kctq6" event={"ID":"b393906b-0a2d-4032-b8c4-0283c219d4c9","Type":"ContainerStarted","Data":"86432e1bb5641de58d8bdfb513cc49b87951e2bd2782361a530318e64bab0f4a"} Jul 21 02:10:53.091784 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:53.091732 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-kctq6" event={"ID":"b393906b-0a2d-4032-b8c4-0283c219d4c9","Type":"ContainerStarted","Data":"04e5d1fddafd98f7fb9ffa70d272aa4d202498199e7e88ad50d0ecba2a7bf9d6"} Jul 21 02:10:53.094004 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:53.093981 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-7llbd" event={"ID":"0cd36c63-e03e-4496-87ae-ffaecb2434c4","Type":"ContainerStarted","Data":"0be1ceff9b999d949f5238854b3169a8af2f86c8c21031509e424cc0bc891dad"} Jul 21 02:10:53.116684 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:53.116641 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-6rb7p" podStartSLOduration=2.056566071 podStartE2EDuration="3.116627271s" totalImagesPullingTime="984.987605ms" totalInitContainerRuntime="75.073595ms" isStatefulPod=true podCreationTimestamp="2026-07-21 02:10:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:10:53.115754986 +0000 UTC m=+87.135248918" watchObservedRunningTime="2026-07-21 02:10:53.116627271 +0000 UTC m=+87.136121175" Jul 21 02:10:53.141947 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:53.141823 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-7llbd" podStartSLOduration=1.701150926 podStartE2EDuration="3.141807547s" totalImagesPullingTime="1.440656621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 02:10:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:10:53.14140237 +0000 UTC m=+87.160896294" watchObservedRunningTime="2026-07-21 02:10:53.141807547 +0000 UTC m=+87.161301452" Jul 21 02:10:53.175250 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:53.175202 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56c5594674-kctq6" podStartSLOduration=1.627262105 podStartE2EDuration="3.175189253s" totalImagesPullingTime="1.547927148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 02:10:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:10:53.174740279 +0000 UTC m=+87.194234182" watchObservedRunningTime="2026-07-21 02:10:53.175189253 +0000 UTC m=+87.194683155" Jul 21 02:10:54.045399 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:54.045366 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-24k47" Jul 21 02:10:54.103767 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:54.103733 2574 generic.go:356] "Generic (PLEG): container finished" podID="2694a3ee-0897-4753-bfee-ff58550df666" containerID="ae4d549addeaff845072cc95c4312d791763717121ba8d61e64b97357f395ca6" exitCode=0 Jul 21 02:10:54.103919 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:54.103838 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2694a3ee-0897-4753-bfee-ff58550df666","Type":"ContainerDied","Data":"ae4d549addeaff845072cc95c4312d791763717121ba8d61e64b97357f395ca6"} Jul 21 02:10:54.714348 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:54.714318 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7dbb746995-2zs6n"] Jul 21 02:10:54.717678 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:54.717653 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7dbb746995-2zs6n" Jul 21 02:10:54.720493 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:54.720467 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 21 02:10:54.720600 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:54.720584 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-8tpdjb6spalun\"" Jul 21 02:10:54.720673 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:54.720595 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 21 02:10:54.720735 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:54.720683 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 21 02:10:54.721509 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:54.721489 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-4c5lm\"" Jul 21 02:10:54.721617 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:54.721546 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 21 02:10:54.728082 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:54.728018 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7dbb746995-2zs6n"] Jul 21 02:10:54.780825 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:54.780800 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b2acc318-de88-402f-a170-17ae1f1b4ff8-metrics-server-audit-profiles\") pod \"metrics-server-7dbb746995-2zs6n\" (UID: \"b2acc318-de88-402f-a170-17ae1f1b4ff8\") " pod="openshift-monitoring/metrics-server-7dbb746995-2zs6n" Jul 21 02:10:54.780967 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:54.780831 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czwq2\" (UniqueName: \"kubernetes.io/projected/b2acc318-de88-402f-a170-17ae1f1b4ff8-kube-api-access-czwq2\") pod \"metrics-server-7dbb746995-2zs6n\" (UID: \"b2acc318-de88-402f-a170-17ae1f1b4ff8\") " pod="openshift-monitoring/metrics-server-7dbb746995-2zs6n" Jul 21 02:10:54.780967 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:54.780852 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b2acc318-de88-402f-a170-17ae1f1b4ff8-secret-metrics-server-tls\") pod \"metrics-server-7dbb746995-2zs6n\" (UID: \"b2acc318-de88-402f-a170-17ae1f1b4ff8\") " pod="openshift-monitoring/metrics-server-7dbb746995-2zs6n" Jul 21 02:10:54.780967 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:54.780874 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b2acc318-de88-402f-a170-17ae1f1b4ff8-audit-log\") pod \"metrics-server-7dbb746995-2zs6n\" (UID: \"b2acc318-de88-402f-a170-17ae1f1b4ff8\") " pod="openshift-monitoring/metrics-server-7dbb746995-2zs6n" Jul 21 02:10:54.781097 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:54.780980 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b2acc318-de88-402f-a170-17ae1f1b4ff8-client-ca-bundle\") pod \"metrics-server-7dbb746995-2zs6n\" (UID: \"b2acc318-de88-402f-a170-17ae1f1b4ff8\") " pod="openshift-monitoring/metrics-server-7dbb746995-2zs6n" Jul 21 02:10:54.781097 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:54.781013 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b2acc318-de88-402f-a170-17ae1f1b4ff8-secret-metrics-server-client-certs\") pod \"metrics-server-7dbb746995-2zs6n\" (UID: \"b2acc318-de88-402f-a170-17ae1f1b4ff8\") " pod="openshift-monitoring/metrics-server-7dbb746995-2zs6n" Jul 21 02:10:54.781097 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:54.781046 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2acc318-de88-402f-a170-17ae1f1b4ff8-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7dbb746995-2zs6n\" (UID: \"b2acc318-de88-402f-a170-17ae1f1b4ff8\") " pod="openshift-monitoring/metrics-server-7dbb746995-2zs6n" Jul 21 02:10:54.882197 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:54.882160 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b2acc318-de88-402f-a170-17ae1f1b4ff8-metrics-server-audit-profiles\") pod \"metrics-server-7dbb746995-2zs6n\" (UID: \"b2acc318-de88-402f-a170-17ae1f1b4ff8\") " pod="openshift-monitoring/metrics-server-7dbb746995-2zs6n" Jul 21 02:10:54.882197 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:54.882196 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czwq2\" (UniqueName: \"kubernetes.io/projected/b2acc318-de88-402f-a170-17ae1f1b4ff8-kube-api-access-czwq2\") pod \"metrics-server-7dbb746995-2zs6n\" (UID: \"b2acc318-de88-402f-a170-17ae1f1b4ff8\") " pod="openshift-monitoring/metrics-server-7dbb746995-2zs6n" Jul 21 02:10:54.882519 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:54.882249 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b2acc318-de88-402f-a170-17ae1f1b4ff8-secret-metrics-server-tls\") pod \"metrics-server-7dbb746995-2zs6n\" (UID: \"b2acc318-de88-402f-a170-17ae1f1b4ff8\") " pod="openshift-monitoring/metrics-server-7dbb746995-2zs6n" Jul 21 02:10:54.882519 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:54.882306 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b2acc318-de88-402f-a170-17ae1f1b4ff8-audit-log\") pod \"metrics-server-7dbb746995-2zs6n\" (UID: \"b2acc318-de88-402f-a170-17ae1f1b4ff8\") " pod="openshift-monitoring/metrics-server-7dbb746995-2zs6n" Jul 21 02:10:54.882519 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:54.882364 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b2acc318-de88-402f-a170-17ae1f1b4ff8-client-ca-bundle\") pod \"metrics-server-7dbb746995-2zs6n\" (UID: \"b2acc318-de88-402f-a170-17ae1f1b4ff8\") " pod="openshift-monitoring/metrics-server-7dbb746995-2zs6n" Jul 21 02:10:54.882519 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:54.882403 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b2acc318-de88-402f-a170-17ae1f1b4ff8-secret-metrics-server-client-certs\") pod \"metrics-server-7dbb746995-2zs6n\" (UID: \"b2acc318-de88-402f-a170-17ae1f1b4ff8\") " pod="openshift-monitoring/metrics-server-7dbb746995-2zs6n" Jul 21 02:10:54.882519 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:54.882442 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2acc318-de88-402f-a170-17ae1f1b4ff8-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7dbb746995-2zs6n\" (UID: \"b2acc318-de88-402f-a170-17ae1f1b4ff8\") " pod="openshift-monitoring/metrics-server-7dbb746995-2zs6n" Jul 21 02:10:54.883026 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:54.882999 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b2acc318-de88-402f-a170-17ae1f1b4ff8-audit-log\") pod \"metrics-server-7dbb746995-2zs6n\" (UID: \"b2acc318-de88-402f-a170-17ae1f1b4ff8\") " pod="openshift-monitoring/metrics-server-7dbb746995-2zs6n" Jul 21 02:10:54.883737 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:54.883714 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b2acc318-de88-402f-a170-17ae1f1b4ff8-metrics-server-audit-profiles\") pod \"metrics-server-7dbb746995-2zs6n\" (UID: \"b2acc318-de88-402f-a170-17ae1f1b4ff8\") " pod="openshift-monitoring/metrics-server-7dbb746995-2zs6n" Jul 21 02:10:54.884316 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:54.884291 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2acc318-de88-402f-a170-17ae1f1b4ff8-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7dbb746995-2zs6n\" (UID: \"b2acc318-de88-402f-a170-17ae1f1b4ff8\") " pod="openshift-monitoring/metrics-server-7dbb746995-2zs6n" Jul 21 02:10:54.885376 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:54.885352 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b2acc318-de88-402f-a170-17ae1f1b4ff8-secret-metrics-server-client-certs\") pod \"metrics-server-7dbb746995-2zs6n\" (UID: \"b2acc318-de88-402f-a170-17ae1f1b4ff8\") " pod="openshift-monitoring/metrics-server-7dbb746995-2zs6n" Jul 21 02:10:54.885468 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:54.885448 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b2acc318-de88-402f-a170-17ae1f1b4ff8-secret-metrics-server-tls\") pod \"metrics-server-7dbb746995-2zs6n\" (UID: \"b2acc318-de88-402f-a170-17ae1f1b4ff8\") " pod="openshift-monitoring/metrics-server-7dbb746995-2zs6n" Jul 21 02:10:54.885511 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:54.885470 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b2acc318-de88-402f-a170-17ae1f1b4ff8-client-ca-bundle\") pod \"metrics-server-7dbb746995-2zs6n\" (UID: \"b2acc318-de88-402f-a170-17ae1f1b4ff8\") " pod="openshift-monitoring/metrics-server-7dbb746995-2zs6n" Jul 21 02:10:54.890540 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:54.890519 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czwq2\" (UniqueName: \"kubernetes.io/projected/b2acc318-de88-402f-a170-17ae1f1b4ff8-kube-api-access-czwq2\") pod \"metrics-server-7dbb746995-2zs6n\" (UID: \"b2acc318-de88-402f-a170-17ae1f1b4ff8\") " pod="openshift-monitoring/metrics-server-7dbb746995-2zs6n" Jul 21 02:10:54.915283 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:54.905409 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-mm7rz"] Jul 21 02:10:54.922300 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:54.922139 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-mm7rz" Jul 21 02:10:54.923949 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:54.923915 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-mm7rz"] Jul 21 02:10:54.925909 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:54.925662 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-qvsn5\"" Jul 21 02:10:54.925909 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:54.925726 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 21 02:10:54.983156 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:54.983087 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/15b83837-cad5-45be-85d5-c8ffa9915341-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-mm7rz\" (UID: \"15b83837-cad5-45be-85d5-c8ffa9915341\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-mm7rz" Jul 21 02:10:54.983156 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:54.983131 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwkjk\" (UniqueName: \"kubernetes.io/projected/15b83837-cad5-45be-85d5-c8ffa9915341-kube-api-access-pwkjk\") pod \"monitoring-plugin-7b7b7bc8bf-mm7rz\" (UID: \"15b83837-cad5-45be-85d5-c8ffa9915341\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-mm7rz" Jul 21 02:10:55.029412 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:55.029384 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7dbb746995-2zs6n" Jul 21 02:10:55.084584 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:55.084544 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/15b83837-cad5-45be-85d5-c8ffa9915341-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-mm7rz\" (UID: \"15b83837-cad5-45be-85d5-c8ffa9915341\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-mm7rz" Jul 21 02:10:55.084967 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:55.084630 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwkjk\" (UniqueName: \"kubernetes.io/projected/15b83837-cad5-45be-85d5-c8ffa9915341-kube-api-access-pwkjk\") pod \"monitoring-plugin-7b7b7bc8bf-mm7rz\" (UID: \"15b83837-cad5-45be-85d5-c8ffa9915341\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-mm7rz" Jul 21 02:10:55.087120 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:55.087086 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/15b83837-cad5-45be-85d5-c8ffa9915341-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-mm7rz\" (UID: \"15b83837-cad5-45be-85d5-c8ffa9915341\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-mm7rz" Jul 21 02:10:55.093804 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:55.093783 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwkjk\" (UniqueName: \"kubernetes.io/projected/15b83837-cad5-45be-85d5-c8ffa9915341-kube-api-access-pwkjk\") pod \"monitoring-plugin-7b7b7bc8bf-mm7rz\" (UID: \"15b83837-cad5-45be-85d5-c8ffa9915341\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-mm7rz" Jul 21 02:10:55.161236 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:55.161207 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7dbb746995-2zs6n"] Jul 21 02:10:55.235159 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:55.235082 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-mm7rz" Jul 21 02:10:55.369207 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:10:55.369173 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2acc318_de88_402f_a170_17ae1f1b4ff8.slice/crio-1d16f1e834710c7459b7c6f6eaf4b4f932e984a74ef29cdc796ab001c5fe3344 WatchSource:0}: Error finding container 1d16f1e834710c7459b7c6f6eaf4b4f932e984a74ef29cdc796ab001c5fe3344: Status 404 returned error can't find the container with id 1d16f1e834710c7459b7c6f6eaf4b4f932e984a74ef29cdc796ab001c5fe3344 Jul 21 02:10:55.542878 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:55.542851 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-mm7rz"] Jul 21 02:10:55.558354 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:10:55.558330 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15b83837_cad5_45be_85d5_c8ffa9915341.slice/crio-3d2e4dd516236a285becb2a3733e1a08bfdb5384a0cc852282ce294257c30831 WatchSource:0}: Error finding container 3d2e4dd516236a285becb2a3733e1a08bfdb5384a0cc852282ce294257c30831: Status 404 returned error can't find the container with id 3d2e4dd516236a285becb2a3733e1a08bfdb5384a0cc852282ce294257c30831 Jul 21 02:10:56.116913 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.116875 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2694a3ee-0897-4753-bfee-ff58550df666","Type":"ContainerStarted","Data":"d76c21e26af634dd1c80480f57b8fcabe22f01b05ff2ce977cc29906b6a63950"} Jul 21 02:10:56.117345 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.116961 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2694a3ee-0897-4753-bfee-ff58550df666","Type":"ContainerStarted","Data":"818fab4e80c6f5bfd7b05e6792b1e2b996a944175d6fce36e7a2e295f4824491"} Jul 21 02:10:56.117345 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.117005 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2694a3ee-0897-4753-bfee-ff58550df666","Type":"ContainerStarted","Data":"cb5a2abe6288e03263d8788e2388acdc1b1b3b982a527ec7043b221f996508a0"} Jul 21 02:10:56.117345 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.117047 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2694a3ee-0897-4753-bfee-ff58550df666","Type":"ContainerStarted","Data":"9a20b7efd192ef3b3776f5529f6d41f6b72ecb135f7b6453579938a402771ba2"} Jul 21 02:10:56.117345 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.117085 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2694a3ee-0897-4753-bfee-ff58550df666","Type":"ContainerStarted","Data":"7b31ecaa94340d4bd1913190e4a0a1c209945aaeaa159f4e4a2cc2bd5c8dcf9d"} Jul 21 02:10:56.118281 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.118241 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-mm7rz" event={"ID":"15b83837-cad5-45be-85d5-c8ffa9915341","Type":"ContainerStarted","Data":"3d2e4dd516236a285becb2a3733e1a08bfdb5384a0cc852282ce294257c30831"} Jul 21 02:10:56.119561 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.119537 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7dbb746995-2zs6n" event={"ID":"b2acc318-de88-402f-a170-17ae1f1b4ff8","Type":"ContainerStarted","Data":"1d16f1e834710c7459b7c6f6eaf4b4f932e984a74ef29cdc796ab001c5fe3344"} Jul 21 02:10:56.343778 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.343736 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-5bc67dcd46-psp69"] Jul 21 02:10:56.349056 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.349024 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-psp69" Jul 21 02:10:56.352914 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.352885 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 21 02:10:56.353046 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.352922 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-x9m8l\"" Jul 21 02:10:56.353139 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.353119 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 21 02:10:56.363037 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.363009 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-psp69"] Jul 21 02:10:56.499215 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.499182 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlcvd\" (UniqueName: \"kubernetes.io/projected/8fabdf69-8196-42e4-9ffc-5f50f6eef7cd-kube-api-access-dlcvd\") pod \"downloads-5bc67dcd46-psp69\" (UID: \"8fabdf69-8196-42e4-9ffc-5f50f6eef7cd\") " pod="openshift-console/downloads-5bc67dcd46-psp69" Jul 21 02:10:56.600930 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.600880 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlcvd\" (UniqueName: \"kubernetes.io/projected/8fabdf69-8196-42e4-9ffc-5f50f6eef7cd-kube-api-access-dlcvd\") pod \"downloads-5bc67dcd46-psp69\" (UID: \"8fabdf69-8196-42e4-9ffc-5f50f6eef7cd\") " pod="openshift-console/downloads-5bc67dcd46-psp69" Jul 21 02:10:56.629855 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.629794 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlcvd\" (UniqueName: \"kubernetes.io/projected/8fabdf69-8196-42e4-9ffc-5f50f6eef7cd-kube-api-access-dlcvd\") pod \"downloads-5bc67dcd46-psp69\" (UID: \"8fabdf69-8196-42e4-9ffc-5f50f6eef7cd\") " pod="openshift-console/downloads-5bc67dcd46-psp69" Jul 21 02:10:56.663567 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.663482 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-psp69" Jul 21 02:10:56.681246 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.680096 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 21 02:10:56.685616 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.685590 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:10:56.692632 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.692600 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 21 02:10:56.692632 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.692626 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 21 02:10:56.692807 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.692608 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 21 02:10:56.692807 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.692600 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 21 02:10:56.692807 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.692700 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 21 02:10:56.692972 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.692909 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 21 02:10:56.693192 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.693147 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 21 02:10:56.693342 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.693241 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-2m0m0vrpk3ssi\"" Jul 21 02:10:56.693671 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.693648 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 21 02:10:56.693801 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.693676 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 21 02:10:56.693801 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.693742 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-f6pls\"" Jul 21 02:10:56.693974 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.693828 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 21 02:10:56.694550 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.694150 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 21 02:10:56.694550 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.694248 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 21 02:10:56.695747 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.695604 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 21 02:10:56.700216 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.700185 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 21 02:10:56.731233 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.731137 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 21 02:10:56.804104 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.804010 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f950068-f8f7-424a-9da8-978105d1e367-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:10:56.804104 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.804049 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0f950068-f8f7-424a-9da8-978105d1e367-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:10:56.804104 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.804091 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0f950068-f8f7-424a-9da8-978105d1e367-config-out\") pod \"prometheus-k8s-0\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:10:56.804337 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.804116 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0f950068-f8f7-424a-9da8-978105d1e367-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:10:56.804337 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.804144 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0f950068-f8f7-424a-9da8-978105d1e367-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:10:56.804337 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.804160 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0f950068-f8f7-424a-9da8-978105d1e367-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:10:56.804337 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.804218 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f950068-f8f7-424a-9da8-978105d1e367-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:10:56.804337 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.804256 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pgqn\" (UniqueName: \"kubernetes.io/projected/0f950068-f8f7-424a-9da8-978105d1e367-kube-api-access-4pgqn\") pod \"prometheus-k8s-0\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:10:56.804337 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.804321 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0f950068-f8f7-424a-9da8-978105d1e367-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:10:56.804585 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.804353 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0f950068-f8f7-424a-9da8-978105d1e367-config\") pod \"prometheus-k8s-0\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:10:56.804585 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.804376 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0f950068-f8f7-424a-9da8-978105d1e367-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:10:56.804585 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.804442 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0f950068-f8f7-424a-9da8-978105d1e367-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:10:56.804585 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.804498 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f950068-f8f7-424a-9da8-978105d1e367-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:10:56.804585 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.804534 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0f950068-f8f7-424a-9da8-978105d1e367-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:10:56.804752 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.804643 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f950068-f8f7-424a-9da8-978105d1e367-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:10:56.804752 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.804682 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0f950068-f8f7-424a-9da8-978105d1e367-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:10:56.804752 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.804729 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0f950068-f8f7-424a-9da8-978105d1e367-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:10:56.804869 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.804769 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0f950068-f8f7-424a-9da8-978105d1e367-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:10:56.804869 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.804786 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0f950068-f8f7-424a-9da8-978105d1e367-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:10:56.804869 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.804802 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0f950068-f8f7-424a-9da8-978105d1e367-web-config\") pod \"prometheus-k8s-0\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:10:56.905323 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.905289 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pgqn\" (UniqueName: \"kubernetes.io/projected/0f950068-f8f7-424a-9da8-978105d1e367-kube-api-access-4pgqn\") pod \"prometheus-k8s-0\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:10:56.905532 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.905345 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0f950068-f8f7-424a-9da8-978105d1e367-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:10:56.905532 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.905391 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0f950068-f8f7-424a-9da8-978105d1e367-config\") pod \"prometheus-k8s-0\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:10:56.905532 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.905416 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0f950068-f8f7-424a-9da8-978105d1e367-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:10:56.905532 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.905440 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0f950068-f8f7-424a-9da8-978105d1e367-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:10:56.905532 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.905464 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f950068-f8f7-424a-9da8-978105d1e367-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:10:56.905532 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.905499 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0f950068-f8f7-424a-9da8-978105d1e367-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:10:56.905532 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.905533 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f950068-f8f7-424a-9da8-978105d1e367-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:10:56.905866 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.905568 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0f950068-f8f7-424a-9da8-978105d1e367-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:10:56.905866 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.905597 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0f950068-f8f7-424a-9da8-978105d1e367-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:10:56.905866 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.905629 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0f950068-f8f7-424a-9da8-978105d1e367-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:10:56.905866 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.905653 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0f950068-f8f7-424a-9da8-978105d1e367-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:10:56.905866 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.905678 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0f950068-f8f7-424a-9da8-978105d1e367-web-config\") pod \"prometheus-k8s-0\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:10:56.905866 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.905708 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f950068-f8f7-424a-9da8-978105d1e367-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:10:56.905866 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.905737 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0f950068-f8f7-424a-9da8-978105d1e367-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:10:56.906183 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.906157 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0f950068-f8f7-424a-9da8-978105d1e367-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:10:56.908338 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.907065 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f950068-f8f7-424a-9da8-978105d1e367-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:10:56.908338 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.907130 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f950068-f8f7-424a-9da8-978105d1e367-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:10:56.908338 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.907432 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0f950068-f8f7-424a-9da8-978105d1e367-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:10:56.908338 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.907878 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0f950068-f8f7-424a-9da8-978105d1e367-config-out\") pod \"prometheus-k8s-0\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:10:56.908338 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.907916 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0f950068-f8f7-424a-9da8-978105d1e367-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:10:56.908338 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.907948 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0f950068-f8f7-424a-9da8-978105d1e367-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:10:56.908338 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.907976 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0f950068-f8f7-424a-9da8-978105d1e367-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:10:56.908338 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.907978 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f950068-f8f7-424a-9da8-978105d1e367-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:10:56.908338 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.908005 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f950068-f8f7-424a-9da8-978105d1e367-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:10:56.908821 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.908690 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f950068-f8f7-424a-9da8-978105d1e367-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:10:56.909881 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.909855 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0f950068-f8f7-424a-9da8-978105d1e367-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:10:56.910255 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.910233 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0f950068-f8f7-424a-9da8-978105d1e367-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:10:56.910402 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.910328 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0f950068-f8f7-424a-9da8-978105d1e367-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:10:56.910663 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.910639 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0f950068-f8f7-424a-9da8-978105d1e367-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:10:56.911421 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.911396 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0f950068-f8f7-424a-9da8-978105d1e367-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:10:56.912114 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.912058 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0f950068-f8f7-424a-9da8-978105d1e367-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:10:56.912796 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.912756 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0f950068-f8f7-424a-9da8-978105d1e367-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:10:56.913324 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.912908 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0f950068-f8f7-424a-9da8-978105d1e367-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:10:56.913324 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.913289 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0f950068-f8f7-424a-9da8-978105d1e367-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:10:56.913863 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.913824 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0f950068-f8f7-424a-9da8-978105d1e367-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:10:56.914372 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.914197 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0f950068-f8f7-424a-9da8-978105d1e367-config-out\") pod \"prometheus-k8s-0\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:10:56.914372 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.914336 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/0f950068-f8f7-424a-9da8-978105d1e367-config\") pod \"prometheus-k8s-0\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:10:56.916053 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.916030 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pgqn\" (UniqueName: \"kubernetes.io/projected/0f950068-f8f7-424a-9da8-978105d1e367-kube-api-access-4pgqn\") pod \"prometheus-k8s-0\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:10:56.917050 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.917012 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0f950068-f8f7-424a-9da8-978105d1e367-web-config\") pod \"prometheus-k8s-0\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:10:56.997581 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:56.997502 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:10:57.505220 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:57.505172 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 21 02:10:57.508840 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:10:57.508817 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f950068_f8f7_424a_9da8_978105d1e367.slice/crio-0985646389f21d814b66741e8fe36de7eb2d13b5bcb0366a0c29a0fce2b1d2f3 WatchSource:0}: Error finding container 0985646389f21d814b66741e8fe36de7eb2d13b5bcb0366a0c29a0fce2b1d2f3: Status 404 returned error can't find the container with id 0985646389f21d814b66741e8fe36de7eb2d13b5bcb0366a0c29a0fce2b1d2f3 Jul 21 02:10:57.512192 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:57.512172 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-psp69"] Jul 21 02:10:57.514335 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:10:57.514311 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fabdf69_8196_42e4_9ffc_5f50f6eef7cd.slice/crio-c4f711600849052296685edc276c7df32b506df3955de6928d329bc2a0cbbe77 WatchSource:0}: Error finding container c4f711600849052296685edc276c7df32b506df3955de6928d329bc2a0cbbe77: Status 404 returned error can't find the container with id c4f711600849052296685edc276c7df32b506df3955de6928d329bc2a0cbbe77 Jul 21 02:10:58.130007 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:58.129963 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-psp69" event={"ID":"8fabdf69-8196-42e4-9ffc-5f50f6eef7cd","Type":"ContainerStarted","Data":"c4f711600849052296685edc276c7df32b506df3955de6928d329bc2a0cbbe77"} Jul 21 02:10:58.131646 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:58.131621 2574 generic.go:356] "Generic (PLEG): container finished" podID="0f950068-f8f7-424a-9da8-978105d1e367" containerID="9edf3aea3263a0d1ac1401f349f9d2285dd6c6f5f2f3e092f77bbd4affbe52ba" exitCode=0 Jul 21 02:10:58.131773 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:58.131714 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f950068-f8f7-424a-9da8-978105d1e367","Type":"ContainerDied","Data":"9edf3aea3263a0d1ac1401f349f9d2285dd6c6f5f2f3e092f77bbd4affbe52ba"} Jul 21 02:10:58.131823 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:58.131783 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f950068-f8f7-424a-9da8-978105d1e367","Type":"ContainerStarted","Data":"0985646389f21d814b66741e8fe36de7eb2d13b5bcb0366a0c29a0fce2b1d2f3"} Jul 21 02:10:58.133689 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:58.133666 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-mm7rz" event={"ID":"15b83837-cad5-45be-85d5-c8ffa9915341","Type":"ContainerStarted","Data":"3130724c1ddcfef53f8259ef258a49a93c0faffe01bc225ef17bf6dc86022f56"} Jul 21 02:10:58.133757 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:58.133725 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-mm7rz" Jul 21 02:10:58.138156 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:58.138132 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2694a3ee-0897-4753-bfee-ff58550df666","Type":"ContainerStarted","Data":"d983dcd2bd4d26fe08596d25e72fdb9cc4f1c4c5e9c7805f33195732bc1b8d3c"} Jul 21 02:10:58.139856 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:58.139835 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-mm7rz" Jul 21 02:10:58.139924 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:58.139874 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7dbb746995-2zs6n" event={"ID":"b2acc318-de88-402f-a170-17ae1f1b4ff8","Type":"ContainerStarted","Data":"05400074af90774b27a64f076bdd831f7a0780ec7f4b4c5c67ff10f9ef1081fb"} Jul 21 02:10:58.179121 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:58.179062 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7dbb746995-2zs6n" podStartSLOduration=2.198618582 podStartE2EDuration="4.179045539s" totalImagesPullingTime="1.980426957s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 02:10:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:10:58.178084478 +0000 UTC m=+92.197578392" watchObservedRunningTime="2026-07-21 02:10:58.179045539 +0000 UTC m=+92.198539507" Jul 21 02:10:58.195179 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:58.195117 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-mm7rz" podStartSLOduration=2.403376739 podStartE2EDuration="4.195095377s" totalImagesPullingTime="1.791718638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 02:10:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:10:58.193851157 +0000 UTC m=+92.213345059" watchObservedRunningTime="2026-07-21 02:10:58.195095377 +0000 UTC m=+92.214589282" Jul 21 02:10:58.229709 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:10:58.229659 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.17769202 podStartE2EDuration="7.229642447s" totalImagesPullingTime="4.033185617s" totalInitContainerRuntime="18.76481ms" isStatefulPod=false podCreationTimestamp="2026-07-21 02:10:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:10:58.227711131 +0000 UTC m=+92.247205053" watchObservedRunningTime="2026-07-21 02:10:58.229642447 +0000 UTC m=+92.249136350" Jul 21 02:11:02.163184 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:11:02.163104 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f950068-f8f7-424a-9da8-978105d1e367","Type":"ContainerStarted","Data":"70408f1726561f0969380425235ac6cffca00e3068b14ec98bc43b706867db78"} Jul 21 02:11:02.163602 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:11:02.163219 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f950068-f8f7-424a-9da8-978105d1e367","Type":"ContainerStarted","Data":"c3908c1ed6bcd63be65ba6d29732376ff55032b44088afdc4a82c668c6b08f3b"} Jul 21 02:11:03.036217 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:11:03.036189 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7bc6c6857c-7rj6n" Jul 21 02:11:05.182050 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:11:05.182012 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f950068-f8f7-424a-9da8-978105d1e367","Type":"ContainerStarted","Data":"af67b4592f00ea2fbf368f75b5606032c0f916e0c97e224ba832e6a380abe3eb"} Jul 21 02:11:05.182598 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:11:05.182089 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f950068-f8f7-424a-9da8-978105d1e367","Type":"ContainerStarted","Data":"35ad05add3e363bd4c278dcac693fb8126f149323fcffe2c9069103294f89949"} Jul 21 02:11:05.182598 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:11:05.182117 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f950068-f8f7-424a-9da8-978105d1e367","Type":"ContainerStarted","Data":"8751472c673329a5da1c64f6f85f32fb53b08fe66b5fd3c61bb2ac08330b7551"} Jul 21 02:11:05.182598 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:11:05.182196 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f950068-f8f7-424a-9da8-978105d1e367","Type":"ContainerStarted","Data":"db57ce3129a039fee6f6bf0b9ed12d22e09a1d89dbf96c265022f8c11b7e440b"} Jul 21 02:11:05.223161 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:11:05.220439 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.25686554 podStartE2EDuration="9.220416712s" totalImagesPullingTime="5.944922834s" totalInitContainerRuntime="18.628338ms" isStatefulPod=false podCreationTimestamp="2026-07-21 02:10:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:11:05.213708866 +0000 UTC m=+99.233202793" watchObservedRunningTime="2026-07-21 02:11:05.220416712 +0000 UTC m=+99.239910619" Jul 21 02:11:06.998135 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:11:06.998093 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:11:13.033370 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:11:13.033340 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qqnx4" Jul 21 02:11:15.030017 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:11:15.029979 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7dbb746995-2zs6n" Jul 21 02:11:15.030410 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:11:15.030079 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7dbb746995-2zs6n" Jul 21 02:11:15.228480 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:11:15.228443 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-psp69" event={"ID":"8fabdf69-8196-42e4-9ffc-5f50f6eef7cd","Type":"ContainerStarted","Data":"ec837a4b3687a4c9d00157654baa6b66cb271fae2097dda030367b8b351d035e"} Jul 21 02:11:15.228691 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:11:15.228670 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-5bc67dcd46-psp69" Jul 21 02:11:15.237402 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:11:15.237377 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-5bc67dcd46-psp69" Jul 21 02:11:15.246158 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:11:15.246112 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-5bc67dcd46-psp69" podStartSLOduration=1.920450355 podStartE2EDuration="19.246099862s" totalImagesPullingTime="17.325649507s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 02:10:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:11:15.244995998 +0000 UTC m=+109.264489933" watchObservedRunningTime="2026-07-21 02:11:15.246099862 +0000 UTC m=+109.265593765" Jul 21 02:11:35.034920 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:11:35.034888 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7dbb746995-2zs6n" Jul 21 02:11:35.038528 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:11:35.038502 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7dbb746995-2zs6n" Jul 21 02:11:56.997954 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:11:56.997912 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:11:57.070819 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:11:57.070777 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:11:57.414158 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:11:57.414092 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:12:10.621398 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:10.621342 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 21 02:12:10.621863 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:10.621710 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2694a3ee-0897-4753-bfee-ff58550df666" containerName="alertmanager" containerID="cri-o://7b31ecaa94340d4bd1913190e4a0a1c209945aaeaa159f4e4a2cc2bd5c8dcf9d" gracePeriod=120 Jul 21 02:12:10.621863 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:10.621788 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2694a3ee-0897-4753-bfee-ff58550df666" containerName="kube-rbac-proxy-web" containerID="cri-o://cb5a2abe6288e03263d8788e2388acdc1b1b3b982a527ec7043b221f996508a0" gracePeriod=120 Jul 21 02:12:10.621863 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:10.621782 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2694a3ee-0897-4753-bfee-ff58550df666" containerName="kube-rbac-proxy-metric" containerID="cri-o://d76c21e26af634dd1c80480f57b8fcabe22f01b05ff2ce977cc29906b6a63950" gracePeriod=120 Jul 21 02:12:10.621863 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:10.621822 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2694a3ee-0897-4753-bfee-ff58550df666" containerName="kube-rbac-proxy" containerID="cri-o://818fab4e80c6f5bfd7b05e6792b1e2b996a944175d6fce36e7a2e295f4824491" gracePeriod=120 Jul 21 02:12:10.622081 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:10.621889 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2694a3ee-0897-4753-bfee-ff58550df666" containerName="prom-label-proxy" containerID="cri-o://d983dcd2bd4d26fe08596d25e72fdb9cc4f1c4c5e9c7805f33195732bc1b8d3c" gracePeriod=120 Jul 21 02:12:10.622081 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:10.621874 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2694a3ee-0897-4753-bfee-ff58550df666" containerName="config-reloader" containerID="cri-o://9a20b7efd192ef3b3776f5529f6d41f6b72ecb135f7b6453579938a402771ba2" gracePeriod=120 Jul 21 02:12:11.456634 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:11.456599 2574 generic.go:356] "Generic (PLEG): container finished" podID="2694a3ee-0897-4753-bfee-ff58550df666" containerID="d983dcd2bd4d26fe08596d25e72fdb9cc4f1c4c5e9c7805f33195732bc1b8d3c" exitCode=0 Jul 21 02:12:11.456634 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:11.456625 2574 generic.go:356] "Generic (PLEG): container finished" podID="2694a3ee-0897-4753-bfee-ff58550df666" containerID="d76c21e26af634dd1c80480f57b8fcabe22f01b05ff2ce977cc29906b6a63950" exitCode=0 Jul 21 02:12:11.456634 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:11.456634 2574 generic.go:356] "Generic (PLEG): container finished" podID="2694a3ee-0897-4753-bfee-ff58550df666" containerID="818fab4e80c6f5bfd7b05e6792b1e2b996a944175d6fce36e7a2e295f4824491" exitCode=0 Jul 21 02:12:11.456634 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:11.456641 2574 generic.go:356] "Generic (PLEG): container finished" podID="2694a3ee-0897-4753-bfee-ff58550df666" containerID="9a20b7efd192ef3b3776f5529f6d41f6b72ecb135f7b6453579938a402771ba2" exitCode=0 Jul 21 02:12:11.456900 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:11.456648 2574 generic.go:356] "Generic (PLEG): container finished" podID="2694a3ee-0897-4753-bfee-ff58550df666" containerID="7b31ecaa94340d4bd1913190e4a0a1c209945aaeaa159f4e4a2cc2bd5c8dcf9d" exitCode=0 Jul 21 02:12:11.456900 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:11.456668 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2694a3ee-0897-4753-bfee-ff58550df666","Type":"ContainerDied","Data":"d983dcd2bd4d26fe08596d25e72fdb9cc4f1c4c5e9c7805f33195732bc1b8d3c"} Jul 21 02:12:11.456900 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:11.456747 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2694a3ee-0897-4753-bfee-ff58550df666","Type":"ContainerDied","Data":"d76c21e26af634dd1c80480f57b8fcabe22f01b05ff2ce977cc29906b6a63950"} Jul 21 02:12:11.456900 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:11.456774 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2694a3ee-0897-4753-bfee-ff58550df666","Type":"ContainerDied","Data":"818fab4e80c6f5bfd7b05e6792b1e2b996a944175d6fce36e7a2e295f4824491"} Jul 21 02:12:11.456900 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:11.456814 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2694a3ee-0897-4753-bfee-ff58550df666","Type":"ContainerDied","Data":"9a20b7efd192ef3b3776f5529f6d41f6b72ecb135f7b6453579938a402771ba2"} Jul 21 02:12:11.456900 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:11.456838 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2694a3ee-0897-4753-bfee-ff58550df666","Type":"ContainerDied","Data":"7b31ecaa94340d4bd1913190e4a0a1c209945aaeaa159f4e4a2cc2bd5c8dcf9d"} Jul 21 02:12:11.857436 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:11.857414 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 21 02:12:11.975133 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:11.975095 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2694a3ee-0897-4753-bfee-ff58550df666-cluster-tls-config\") pod \"2694a3ee-0897-4753-bfee-ff58550df666\" (UID: \"2694a3ee-0897-4753-bfee-ff58550df666\") " Jul 21 02:12:11.975133 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:11.975137 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2694a3ee-0897-4753-bfee-ff58550df666-secret-alertmanager-main-tls\") pod \"2694a3ee-0897-4753-bfee-ff58550df666\" (UID: \"2694a3ee-0897-4753-bfee-ff58550df666\") " Jul 21 02:12:11.975413 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:11.975172 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2694a3ee-0897-4753-bfee-ff58550df666-secret-alertmanager-kube-rbac-proxy-web\") pod \"2694a3ee-0897-4753-bfee-ff58550df666\" (UID: \"2694a3ee-0897-4753-bfee-ff58550df666\") " Jul 21 02:12:11.975413 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:11.975203 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2694a3ee-0897-4753-bfee-ff58550df666-alertmanager-main-db\") pod \"2694a3ee-0897-4753-bfee-ff58550df666\" (UID: \"2694a3ee-0897-4753-bfee-ff58550df666\") " Jul 21 02:12:11.975413 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:11.975227 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2694a3ee-0897-4753-bfee-ff58550df666-metrics-client-ca\") pod \"2694a3ee-0897-4753-bfee-ff58550df666\" (UID: \"2694a3ee-0897-4753-bfee-ff58550df666\") " Jul 21 02:12:11.975413 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:11.975326 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2694a3ee-0897-4753-bfee-ff58550df666-alertmanager-trusted-ca-bundle\") pod \"2694a3ee-0897-4753-bfee-ff58550df666\" (UID: \"2694a3ee-0897-4753-bfee-ff58550df666\") " Jul 21 02:12:11.975413 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:11.975379 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mn9kj\" (UniqueName: \"kubernetes.io/projected/2694a3ee-0897-4753-bfee-ff58550df666-kube-api-access-mn9kj\") pod \"2694a3ee-0897-4753-bfee-ff58550df666\" (UID: \"2694a3ee-0897-4753-bfee-ff58550df666\") " Jul 21 02:12:11.975659 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:11.975492 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2694a3ee-0897-4753-bfee-ff58550df666-secret-alertmanager-kube-rbac-proxy-metric\") pod \"2694a3ee-0897-4753-bfee-ff58550df666\" (UID: \"2694a3ee-0897-4753-bfee-ff58550df666\") " Jul 21 02:12:11.975659 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:11.975529 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2694a3ee-0897-4753-bfee-ff58550df666-secret-alertmanager-kube-rbac-proxy\") pod \"2694a3ee-0897-4753-bfee-ff58550df666\" (UID: \"2694a3ee-0897-4753-bfee-ff58550df666\") " Jul 21 02:12:11.975659 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:11.975543 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2694a3ee-0897-4753-bfee-ff58550df666-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "2694a3ee-0897-4753-bfee-ff58550df666" (UID: "2694a3ee-0897-4753-bfee-ff58550df666"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 02:12:11.975659 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:11.975574 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2694a3ee-0897-4753-bfee-ff58550df666-config-out\") pod \"2694a3ee-0897-4753-bfee-ff58550df666\" (UID: \"2694a3ee-0897-4753-bfee-ff58550df666\") " Jul 21 02:12:11.975659 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:11.975652 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2694a3ee-0897-4753-bfee-ff58550df666-web-config\") pod \"2694a3ee-0897-4753-bfee-ff58550df666\" (UID: \"2694a3ee-0897-4753-bfee-ff58550df666\") " Jul 21 02:12:11.975898 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:11.975684 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2694a3ee-0897-4753-bfee-ff58550df666-tls-assets\") pod \"2694a3ee-0897-4753-bfee-ff58550df666\" (UID: \"2694a3ee-0897-4753-bfee-ff58550df666\") " Jul 21 02:12:11.975898 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:11.975726 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2694a3ee-0897-4753-bfee-ff58550df666-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "2694a3ee-0897-4753-bfee-ff58550df666" (UID: "2694a3ee-0897-4753-bfee-ff58550df666"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 02:12:11.975898 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:11.975742 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2694a3ee-0897-4753-bfee-ff58550df666-config-volume\") pod \"2694a3ee-0897-4753-bfee-ff58550df666\" (UID: \"2694a3ee-0897-4753-bfee-ff58550df666\") " Jul 21 02:12:11.976061 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:11.975948 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2694a3ee-0897-4753-bfee-ff58550df666-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "2694a3ee-0897-4753-bfee-ff58550df666" (UID: "2694a3ee-0897-4753-bfee-ff58550df666"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 02:12:11.976168 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:11.976146 2574 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2694a3ee-0897-4753-bfee-ff58550df666-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-141-37.ec2.internal\" DevicePath \"\"" Jul 21 02:12:11.976237 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:11.976174 2574 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2694a3ee-0897-4753-bfee-ff58550df666-alertmanager-main-db\") on node \"ip-10-0-141-37.ec2.internal\" DevicePath \"\"" Jul 21 02:12:11.976237 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:11.976190 2574 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2694a3ee-0897-4753-bfee-ff58550df666-metrics-client-ca\") on node \"ip-10-0-141-37.ec2.internal\" DevicePath \"\"" Jul 21 02:12:11.978379 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:11.978342 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2694a3ee-0897-4753-bfee-ff58550df666-kube-api-access-mn9kj" (OuterVolumeSpecName: "kube-api-access-mn9kj") pod "2694a3ee-0897-4753-bfee-ff58550df666" (UID: "2694a3ee-0897-4753-bfee-ff58550df666"). InnerVolumeSpecName "kube-api-access-mn9kj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 02:12:11.978669 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:11.978649 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2694a3ee-0897-4753-bfee-ff58550df666-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "2694a3ee-0897-4753-bfee-ff58550df666" (UID: "2694a3ee-0897-4753-bfee-ff58550df666"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 02:12:11.978795 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:11.978774 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2694a3ee-0897-4753-bfee-ff58550df666-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "2694a3ee-0897-4753-bfee-ff58550df666" (UID: "2694a3ee-0897-4753-bfee-ff58550df666"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 02:12:11.979207 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:11.979181 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2694a3ee-0897-4753-bfee-ff58550df666-config-out" (OuterVolumeSpecName: "config-out") pod "2694a3ee-0897-4753-bfee-ff58550df666" (UID: "2694a3ee-0897-4753-bfee-ff58550df666"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 02:12:11.979207 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:11.979203 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2694a3ee-0897-4753-bfee-ff58550df666-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "2694a3ee-0897-4753-bfee-ff58550df666" (UID: "2694a3ee-0897-4753-bfee-ff58550df666"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 02:12:11.979392 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:11.979227 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2694a3ee-0897-4753-bfee-ff58550df666-config-volume" (OuterVolumeSpecName: "config-volume") pod "2694a3ee-0897-4753-bfee-ff58550df666" (UID: "2694a3ee-0897-4753-bfee-ff58550df666"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 02:12:11.979598 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:11.979565 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2694a3ee-0897-4753-bfee-ff58550df666-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "2694a3ee-0897-4753-bfee-ff58550df666" (UID: "2694a3ee-0897-4753-bfee-ff58550df666"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 02:12:11.980231 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:11.980203 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2694a3ee-0897-4753-bfee-ff58550df666-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "2694a3ee-0897-4753-bfee-ff58550df666" (UID: "2694a3ee-0897-4753-bfee-ff58550df666"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 02:12:11.982563 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:11.982498 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2694a3ee-0897-4753-bfee-ff58550df666-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "2694a3ee-0897-4753-bfee-ff58550df666" (UID: "2694a3ee-0897-4753-bfee-ff58550df666"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 02:12:11.988922 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:11.988897 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2694a3ee-0897-4753-bfee-ff58550df666-web-config" (OuterVolumeSpecName: "web-config") pod "2694a3ee-0897-4753-bfee-ff58550df666" (UID: "2694a3ee-0897-4753-bfee-ff58550df666"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 02:12:12.077108 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:12.077063 2574 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2694a3ee-0897-4753-bfee-ff58550df666-config-out\") on node \"ip-10-0-141-37.ec2.internal\" DevicePath \"\"" Jul 21 02:12:12.077108 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:12.077104 2574 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2694a3ee-0897-4753-bfee-ff58550df666-web-config\") on node \"ip-10-0-141-37.ec2.internal\" DevicePath \"\"" Jul 21 02:12:12.077108 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:12.077113 2574 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2694a3ee-0897-4753-bfee-ff58550df666-tls-assets\") on node \"ip-10-0-141-37.ec2.internal\" DevicePath \"\"" Jul 21 02:12:12.077108 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:12.077120 2574 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2694a3ee-0897-4753-bfee-ff58550df666-config-volume\") on node \"ip-10-0-141-37.ec2.internal\" DevicePath \"\"" Jul 21 02:12:12.077108 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:12.077128 2574 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2694a3ee-0897-4753-bfee-ff58550df666-cluster-tls-config\") on node \"ip-10-0-141-37.ec2.internal\" DevicePath \"\"" Jul 21 02:12:12.077416 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:12.077137 2574 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2694a3ee-0897-4753-bfee-ff58550df666-secret-alertmanager-main-tls\") on node \"ip-10-0-141-37.ec2.internal\" DevicePath \"\"" Jul 21 02:12:12.077416 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:12.077146 2574 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2694a3ee-0897-4753-bfee-ff58550df666-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-141-37.ec2.internal\" DevicePath \"\"" Jul 21 02:12:12.077416 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:12.077154 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mn9kj\" (UniqueName: \"kubernetes.io/projected/2694a3ee-0897-4753-bfee-ff58550df666-kube-api-access-mn9kj\") on node \"ip-10-0-141-37.ec2.internal\" DevicePath \"\"" Jul 21 02:12:12.077416 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:12.077162 2574 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2694a3ee-0897-4753-bfee-ff58550df666-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-141-37.ec2.internal\" DevicePath \"\"" Jul 21 02:12:12.077416 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:12.077172 2574 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2694a3ee-0897-4753-bfee-ff58550df666-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-141-37.ec2.internal\" DevicePath \"\"" Jul 21 02:12:12.463304 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:12.463253 2574 generic.go:356] "Generic (PLEG): container finished" podID="2694a3ee-0897-4753-bfee-ff58550df666" containerID="cb5a2abe6288e03263d8788e2388acdc1b1b3b982a527ec7043b221f996508a0" exitCode=0 Jul 21 02:12:12.463304 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:12.463295 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2694a3ee-0897-4753-bfee-ff58550df666","Type":"ContainerDied","Data":"cb5a2abe6288e03263d8788e2388acdc1b1b3b982a527ec7043b221f996508a0"} Jul 21 02:12:12.463538 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:12.463361 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 21 02:12:12.463538 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:12.463378 2574 scope.go:117] "RemoveContainer" containerID="d983dcd2bd4d26fe08596d25e72fdb9cc4f1c4c5e9c7805f33195732bc1b8d3c" Jul 21 02:12:12.463538 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:12.463365 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2694a3ee-0897-4753-bfee-ff58550df666","Type":"ContainerDied","Data":"3f704f290a811d21b14b4e999ccaec48bbeb5209d866aedb8e73f62d6a0fd521"} Jul 21 02:12:12.471752 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:12.471732 2574 scope.go:117] "RemoveContainer" containerID="d76c21e26af634dd1c80480f57b8fcabe22f01b05ff2ce977cc29906b6a63950" Jul 21 02:12:12.478337 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:12.478319 2574 scope.go:117] "RemoveContainer" containerID="818fab4e80c6f5bfd7b05e6792b1e2b996a944175d6fce36e7a2e295f4824491" Jul 21 02:12:12.484754 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:12.484738 2574 scope.go:117] "RemoveContainer" containerID="cb5a2abe6288e03263d8788e2388acdc1b1b3b982a527ec7043b221f996508a0" Jul 21 02:12:12.489061 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:12.489041 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 21 02:12:12.492730 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:12.492713 2574 scope.go:117] "RemoveContainer" containerID="9a20b7efd192ef3b3776f5529f6d41f6b72ecb135f7b6453579938a402771ba2" Jul 21 02:12:12.492805 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:12.492789 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 21 02:12:12.499302 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:12.499287 2574 scope.go:117] "RemoveContainer" containerID="7b31ecaa94340d4bd1913190e4a0a1c209945aaeaa159f4e4a2cc2bd5c8dcf9d" Jul 21 02:12:12.505755 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:12.505739 2574 scope.go:117] "RemoveContainer" containerID="ae4d549addeaff845072cc95c4312d791763717121ba8d61e64b97357f395ca6" Jul 21 02:12:12.512520 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:12.512500 2574 scope.go:117] "RemoveContainer" containerID="d983dcd2bd4d26fe08596d25e72fdb9cc4f1c4c5e9c7805f33195732bc1b8d3c" Jul 21 02:12:12.512776 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:12:12.512756 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d983dcd2bd4d26fe08596d25e72fdb9cc4f1c4c5e9c7805f33195732bc1b8d3c\": container with ID starting with d983dcd2bd4d26fe08596d25e72fdb9cc4f1c4c5e9c7805f33195732bc1b8d3c not found: ID does not exist" containerID="d983dcd2bd4d26fe08596d25e72fdb9cc4f1c4c5e9c7805f33195732bc1b8d3c" Jul 21 02:12:12.512840 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:12.512784 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d983dcd2bd4d26fe08596d25e72fdb9cc4f1c4c5e9c7805f33195732bc1b8d3c"} err="failed to get container status \"d983dcd2bd4d26fe08596d25e72fdb9cc4f1c4c5e9c7805f33195732bc1b8d3c\": rpc error: code = NotFound desc = could not find container \"d983dcd2bd4d26fe08596d25e72fdb9cc4f1c4c5e9c7805f33195732bc1b8d3c\": container with ID starting with d983dcd2bd4d26fe08596d25e72fdb9cc4f1c4c5e9c7805f33195732bc1b8d3c not found: ID does not exist" Jul 21 02:12:12.512840 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:12.512818 2574 scope.go:117] "RemoveContainer" containerID="d76c21e26af634dd1c80480f57b8fcabe22f01b05ff2ce977cc29906b6a63950" Jul 21 02:12:12.513056 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:12:12.513041 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d76c21e26af634dd1c80480f57b8fcabe22f01b05ff2ce977cc29906b6a63950\": container with ID starting with d76c21e26af634dd1c80480f57b8fcabe22f01b05ff2ce977cc29906b6a63950 not found: ID does not exist" containerID="d76c21e26af634dd1c80480f57b8fcabe22f01b05ff2ce977cc29906b6a63950" Jul 21 02:12:12.513111 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:12.513062 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d76c21e26af634dd1c80480f57b8fcabe22f01b05ff2ce977cc29906b6a63950"} err="failed to get container status \"d76c21e26af634dd1c80480f57b8fcabe22f01b05ff2ce977cc29906b6a63950\": rpc error: code = NotFound desc = could not find container \"d76c21e26af634dd1c80480f57b8fcabe22f01b05ff2ce977cc29906b6a63950\": container with ID starting with d76c21e26af634dd1c80480f57b8fcabe22f01b05ff2ce977cc29906b6a63950 not found: ID does not exist" Jul 21 02:12:12.513111 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:12.513079 2574 scope.go:117] "RemoveContainer" containerID="818fab4e80c6f5bfd7b05e6792b1e2b996a944175d6fce36e7a2e295f4824491" Jul 21 02:12:12.513307 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:12:12.513291 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"818fab4e80c6f5bfd7b05e6792b1e2b996a944175d6fce36e7a2e295f4824491\": container with ID starting with 818fab4e80c6f5bfd7b05e6792b1e2b996a944175d6fce36e7a2e295f4824491 not found: ID does not exist" containerID="818fab4e80c6f5bfd7b05e6792b1e2b996a944175d6fce36e7a2e295f4824491" Jul 21 02:12:12.513356 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:12.513312 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"818fab4e80c6f5bfd7b05e6792b1e2b996a944175d6fce36e7a2e295f4824491"} err="failed to get container status \"818fab4e80c6f5bfd7b05e6792b1e2b996a944175d6fce36e7a2e295f4824491\": rpc error: code = NotFound desc = could not find container \"818fab4e80c6f5bfd7b05e6792b1e2b996a944175d6fce36e7a2e295f4824491\": container with ID starting with 818fab4e80c6f5bfd7b05e6792b1e2b996a944175d6fce36e7a2e295f4824491 not found: ID does not exist" Jul 21 02:12:12.513356 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:12.513326 2574 scope.go:117] "RemoveContainer" containerID="cb5a2abe6288e03263d8788e2388acdc1b1b3b982a527ec7043b221f996508a0" Jul 21 02:12:12.513549 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:12:12.513532 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb5a2abe6288e03263d8788e2388acdc1b1b3b982a527ec7043b221f996508a0\": container with ID starting with cb5a2abe6288e03263d8788e2388acdc1b1b3b982a527ec7043b221f996508a0 not found: ID does not exist" containerID="cb5a2abe6288e03263d8788e2388acdc1b1b3b982a527ec7043b221f996508a0" Jul 21 02:12:12.513589 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:12.513555 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb5a2abe6288e03263d8788e2388acdc1b1b3b982a527ec7043b221f996508a0"} err="failed to get container status \"cb5a2abe6288e03263d8788e2388acdc1b1b3b982a527ec7043b221f996508a0\": rpc error: code = NotFound desc = could not find container \"cb5a2abe6288e03263d8788e2388acdc1b1b3b982a527ec7043b221f996508a0\": container with ID starting with cb5a2abe6288e03263d8788e2388acdc1b1b3b982a527ec7043b221f996508a0 not found: ID does not exist" Jul 21 02:12:12.513589 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:12.513574 2574 scope.go:117] "RemoveContainer" containerID="9a20b7efd192ef3b3776f5529f6d41f6b72ecb135f7b6453579938a402771ba2" Jul 21 02:12:12.513812 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:12:12.513786 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a20b7efd192ef3b3776f5529f6d41f6b72ecb135f7b6453579938a402771ba2\": container with ID starting with 9a20b7efd192ef3b3776f5529f6d41f6b72ecb135f7b6453579938a402771ba2 not found: ID does not exist" containerID="9a20b7efd192ef3b3776f5529f6d41f6b72ecb135f7b6453579938a402771ba2" Jul 21 02:12:12.513881 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:12.513812 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a20b7efd192ef3b3776f5529f6d41f6b72ecb135f7b6453579938a402771ba2"} err="failed to get container status \"9a20b7efd192ef3b3776f5529f6d41f6b72ecb135f7b6453579938a402771ba2\": rpc error: code = NotFound desc = could not find container \"9a20b7efd192ef3b3776f5529f6d41f6b72ecb135f7b6453579938a402771ba2\": container with ID starting with 9a20b7efd192ef3b3776f5529f6d41f6b72ecb135f7b6453579938a402771ba2 not found: ID does not exist" Jul 21 02:12:12.513881 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:12.513832 2574 scope.go:117] "RemoveContainer" containerID="7b31ecaa94340d4bd1913190e4a0a1c209945aaeaa159f4e4a2cc2bd5c8dcf9d" Jul 21 02:12:12.514074 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:12:12.514058 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b31ecaa94340d4bd1913190e4a0a1c209945aaeaa159f4e4a2cc2bd5c8dcf9d\": container with ID starting with 7b31ecaa94340d4bd1913190e4a0a1c209945aaeaa159f4e4a2cc2bd5c8dcf9d not found: ID does not exist" containerID="7b31ecaa94340d4bd1913190e4a0a1c209945aaeaa159f4e4a2cc2bd5c8dcf9d" Jul 21 02:12:12.514114 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:12.514081 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b31ecaa94340d4bd1913190e4a0a1c209945aaeaa159f4e4a2cc2bd5c8dcf9d"} err="failed to get container status \"7b31ecaa94340d4bd1913190e4a0a1c209945aaeaa159f4e4a2cc2bd5c8dcf9d\": rpc error: code = NotFound desc = could not find container \"7b31ecaa94340d4bd1913190e4a0a1c209945aaeaa159f4e4a2cc2bd5c8dcf9d\": container with ID starting with 7b31ecaa94340d4bd1913190e4a0a1c209945aaeaa159f4e4a2cc2bd5c8dcf9d not found: ID does not exist" Jul 21 02:12:12.514114 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:12.514094 2574 scope.go:117] "RemoveContainer" containerID="ae4d549addeaff845072cc95c4312d791763717121ba8d61e64b97357f395ca6" Jul 21 02:12:12.514353 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:12:12.514327 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae4d549addeaff845072cc95c4312d791763717121ba8d61e64b97357f395ca6\": container with ID starting with ae4d549addeaff845072cc95c4312d791763717121ba8d61e64b97357f395ca6 not found: ID does not exist" containerID="ae4d549addeaff845072cc95c4312d791763717121ba8d61e64b97357f395ca6" Jul 21 02:12:12.514404 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:12.514349 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae4d549addeaff845072cc95c4312d791763717121ba8d61e64b97357f395ca6"} err="failed to get container status \"ae4d549addeaff845072cc95c4312d791763717121ba8d61e64b97357f395ca6\": rpc error: code = NotFound desc = could not find container \"ae4d549addeaff845072cc95c4312d791763717121ba8d61e64b97357f395ca6\": container with ID starting with ae4d549addeaff845072cc95c4312d791763717121ba8d61e64b97357f395ca6 not found: ID does not exist" Jul 21 02:12:14.484193 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:14.484158 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2694a3ee-0897-4753-bfee-ff58550df666" path="/var/lib/kubelet/pods/2694a3ee-0897-4753-bfee-ff58550df666/volumes" Jul 21 02:12:14.838848 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:14.838754 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 21 02:12:14.839124 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:14.839097 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0f950068-f8f7-424a-9da8-978105d1e367" containerName="prometheus" containerID="cri-o://c3908c1ed6bcd63be65ba6d29732376ff55032b44088afdc4a82c668c6b08f3b" gracePeriod=600 Jul 21 02:12:14.839226 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:14.839120 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0f950068-f8f7-424a-9da8-978105d1e367" containerName="kube-rbac-proxy" containerID="cri-o://35ad05add3e363bd4c278dcac693fb8126f149323fcffe2c9069103294f89949" gracePeriod=600 Jul 21 02:12:14.839226 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:14.839133 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0f950068-f8f7-424a-9da8-978105d1e367" containerName="thanos-sidecar" containerID="cri-o://db57ce3129a039fee6f6bf0b9ed12d22e09a1d89dbf96c265022f8c11b7e440b" gracePeriod=600 Jul 21 02:12:14.839226 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:14.839158 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0f950068-f8f7-424a-9da8-978105d1e367" containerName="kube-rbac-proxy-web" containerID="cri-o://8751472c673329a5da1c64f6f85f32fb53b08fe66b5fd3c61bb2ac08330b7551" gracePeriod=600 Jul 21 02:12:14.839226 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:14.839190 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0f950068-f8f7-424a-9da8-978105d1e367" containerName="kube-rbac-proxy-thanos" containerID="cri-o://af67b4592f00ea2fbf368f75b5606032c0f916e0c97e224ba832e6a380abe3eb" gracePeriod=600 Jul 21 02:12:14.839500 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:14.839134 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0f950068-f8f7-424a-9da8-978105d1e367" containerName="config-reloader" containerID="cri-o://70408f1726561f0969380425235ac6cffca00e3068b14ec98bc43b706867db78" gracePeriod=600 Jul 21 02:12:15.080376 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.080351 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:12:15.201991 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.201965 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0f950068-f8f7-424a-9da8-978105d1e367-config-out\") pod \"0f950068-f8f7-424a-9da8-978105d1e367\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " Jul 21 02:12:15.202160 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.202001 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f950068-f8f7-424a-9da8-978105d1e367-configmap-metrics-client-ca\") pod \"0f950068-f8f7-424a-9da8-978105d1e367\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " Jul 21 02:12:15.202160 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.202024 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0f950068-f8f7-424a-9da8-978105d1e367-web-config\") pod \"0f950068-f8f7-424a-9da8-978105d1e367\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " Jul 21 02:12:15.202160 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.202044 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0f950068-f8f7-424a-9da8-978105d1e367-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"0f950068-f8f7-424a-9da8-978105d1e367\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " Jul 21 02:12:15.202160 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.202067 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0f950068-f8f7-424a-9da8-978105d1e367-secret-prometheus-k8s-tls\") pod \"0f950068-f8f7-424a-9da8-978105d1e367\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " Jul 21 02:12:15.202160 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.202104 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0f950068-f8f7-424a-9da8-978105d1e367-secret-grpc-tls\") pod \"0f950068-f8f7-424a-9da8-978105d1e367\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " Jul 21 02:12:15.202160 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.202135 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0f950068-f8f7-424a-9da8-978105d1e367-thanos-prometheus-http-client-file\") pod \"0f950068-f8f7-424a-9da8-978105d1e367\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " Jul 21 02:12:15.202497 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.202187 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f950068-f8f7-424a-9da8-978105d1e367-configmap-serving-certs-ca-bundle\") pod \"0f950068-f8f7-424a-9da8-978105d1e367\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " Jul 21 02:12:15.202497 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.202219 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0f950068-f8f7-424a-9da8-978105d1e367-prometheus-k8s-rulefiles-0\") pod \"0f950068-f8f7-424a-9da8-978105d1e367\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " Jul 21 02:12:15.202497 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.202246 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4pgqn\" (UniqueName: \"kubernetes.io/projected/0f950068-f8f7-424a-9da8-978105d1e367-kube-api-access-4pgqn\") pod \"0f950068-f8f7-424a-9da8-978105d1e367\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " Jul 21 02:12:15.202497 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.202302 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0f950068-f8f7-424a-9da8-978105d1e367-secret-kube-rbac-proxy\") pod \"0f950068-f8f7-424a-9da8-978105d1e367\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " Jul 21 02:12:15.202497 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.202327 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0f950068-f8f7-424a-9da8-978105d1e367-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"0f950068-f8f7-424a-9da8-978105d1e367\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " Jul 21 02:12:15.202497 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.202358 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f950068-f8f7-424a-9da8-978105d1e367-prometheus-trusted-ca-bundle\") pod \"0f950068-f8f7-424a-9da8-978105d1e367\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " Jul 21 02:12:15.202497 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.202392 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f950068-f8f7-424a-9da8-978105d1e367-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "0f950068-f8f7-424a-9da8-978105d1e367" (UID: "0f950068-f8f7-424a-9da8-978105d1e367"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 02:12:15.202836 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.202675 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f950068-f8f7-424a-9da8-978105d1e367-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "0f950068-f8f7-424a-9da8-978105d1e367" (UID: "0f950068-f8f7-424a-9da8-978105d1e367"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 02:12:15.202836 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.202682 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f950068-f8f7-424a-9da8-978105d1e367-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "0f950068-f8f7-424a-9da8-978105d1e367" (UID: "0f950068-f8f7-424a-9da8-978105d1e367"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 02:12:15.203584 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.203556 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f950068-f8f7-424a-9da8-978105d1e367-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "0f950068-f8f7-424a-9da8-978105d1e367" (UID: "0f950068-f8f7-424a-9da8-978105d1e367"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 02:12:15.204636 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.204606 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f950068-f8f7-424a-9da8-978105d1e367-config-out" (OuterVolumeSpecName: "config-out") pod "0f950068-f8f7-424a-9da8-978105d1e367" (UID: "0f950068-f8f7-424a-9da8-978105d1e367"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 02:12:15.204736 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.204641 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f950068-f8f7-424a-9da8-978105d1e367-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "0f950068-f8f7-424a-9da8-978105d1e367" (UID: "0f950068-f8f7-424a-9da8-978105d1e367"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 02:12:15.204736 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.204705 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0f950068-f8f7-424a-9da8-978105d1e367-secret-metrics-client-certs\") pod \"0f950068-f8f7-424a-9da8-978105d1e367\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " Jul 21 02:12:15.204861 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.204744 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0f950068-f8f7-424a-9da8-978105d1e367-prometheus-k8s-rulefiles-2\") pod \"0f950068-f8f7-424a-9da8-978105d1e367\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " Jul 21 02:12:15.204861 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.204786 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f950068-f8f7-424a-9da8-978105d1e367-configmap-kubelet-serving-ca-bundle\") pod \"0f950068-f8f7-424a-9da8-978105d1e367\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " Jul 21 02:12:15.204861 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.204821 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0f950068-f8f7-424a-9da8-978105d1e367-prometheus-k8s-rulefiles-1\") pod \"0f950068-f8f7-424a-9da8-978105d1e367\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " Jul 21 02:12:15.204861 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.204850 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0f950068-f8f7-424a-9da8-978105d1e367-tls-assets\") pod \"0f950068-f8f7-424a-9da8-978105d1e367\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " Jul 21 02:12:15.205066 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.204888 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0f950068-f8f7-424a-9da8-978105d1e367-config\") pod \"0f950068-f8f7-424a-9da8-978105d1e367\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " Jul 21 02:12:15.205066 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.204912 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0f950068-f8f7-424a-9da8-978105d1e367-prometheus-k8s-db\") pod \"0f950068-f8f7-424a-9da8-978105d1e367\" (UID: \"0f950068-f8f7-424a-9da8-978105d1e367\") " Jul 21 02:12:15.205066 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.205051 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f950068-f8f7-424a-9da8-978105d1e367-kube-api-access-4pgqn" (OuterVolumeSpecName: "kube-api-access-4pgqn") pod "0f950068-f8f7-424a-9da8-978105d1e367" (UID: "0f950068-f8f7-424a-9da8-978105d1e367"). InnerVolumeSpecName "kube-api-access-4pgqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 02:12:15.207134 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.205181 2574 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0f950068-f8f7-424a-9da8-978105d1e367-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-141-37.ec2.internal\" DevicePath \"\"" Jul 21 02:12:15.207134 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.205225 2574 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f950068-f8f7-424a-9da8-978105d1e367-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-141-37.ec2.internal\" DevicePath \"\"" Jul 21 02:12:15.207134 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.205240 2574 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0f950068-f8f7-424a-9da8-978105d1e367-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-141-37.ec2.internal\" DevicePath \"\"" Jul 21 02:12:15.207134 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.205254 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pgqn\" (UniqueName: \"kubernetes.io/projected/0f950068-f8f7-424a-9da8-978105d1e367-kube-api-access-4pgqn\") on node \"ip-10-0-141-37.ec2.internal\" DevicePath \"\"" Jul 21 02:12:15.207134 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.205285 2574 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f950068-f8f7-424a-9da8-978105d1e367-prometheus-trusted-ca-bundle\") on node \"ip-10-0-141-37.ec2.internal\" DevicePath \"\"" Jul 21 02:12:15.207134 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.205298 2574 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0f950068-f8f7-424a-9da8-978105d1e367-config-out\") on node \"ip-10-0-141-37.ec2.internal\" DevicePath \"\"" Jul 21 02:12:15.207134 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.205310 2574 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f950068-f8f7-424a-9da8-978105d1e367-configmap-metrics-client-ca\") on node \"ip-10-0-141-37.ec2.internal\" DevicePath \"\"" Jul 21 02:12:15.207134 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.205794 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f950068-f8f7-424a-9da8-978105d1e367-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "0f950068-f8f7-424a-9da8-978105d1e367" (UID: "0f950068-f8f7-424a-9da8-978105d1e367"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 02:12:15.207134 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.206062 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f950068-f8f7-424a-9da8-978105d1e367-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "0f950068-f8f7-424a-9da8-978105d1e367" (UID: "0f950068-f8f7-424a-9da8-978105d1e367"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 02:12:15.207134 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.206435 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f950068-f8f7-424a-9da8-978105d1e367-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "0f950068-f8f7-424a-9da8-978105d1e367" (UID: "0f950068-f8f7-424a-9da8-978105d1e367"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 02:12:15.208755 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.208719 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f950068-f8f7-424a-9da8-978105d1e367-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "0f950068-f8f7-424a-9da8-978105d1e367" (UID: "0f950068-f8f7-424a-9da8-978105d1e367"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 02:12:15.210820 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.210783 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f950068-f8f7-424a-9da8-978105d1e367-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "0f950068-f8f7-424a-9da8-978105d1e367" (UID: "0f950068-f8f7-424a-9da8-978105d1e367"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 02:12:15.210918 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.210818 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f950068-f8f7-424a-9da8-978105d1e367-config" (OuterVolumeSpecName: "config") pod "0f950068-f8f7-424a-9da8-978105d1e367" (UID: "0f950068-f8f7-424a-9da8-978105d1e367"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 02:12:15.211342 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.211316 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f950068-f8f7-424a-9da8-978105d1e367-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "0f950068-f8f7-424a-9da8-978105d1e367" (UID: "0f950068-f8f7-424a-9da8-978105d1e367"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 02:12:15.211643 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.211611 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f950068-f8f7-424a-9da8-978105d1e367-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "0f950068-f8f7-424a-9da8-978105d1e367" (UID: "0f950068-f8f7-424a-9da8-978105d1e367"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 02:12:15.211921 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.211887 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f950068-f8f7-424a-9da8-978105d1e367-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "0f950068-f8f7-424a-9da8-978105d1e367" (UID: "0f950068-f8f7-424a-9da8-978105d1e367"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 02:12:15.212398 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.212372 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f950068-f8f7-424a-9da8-978105d1e367-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "0f950068-f8f7-424a-9da8-978105d1e367" (UID: "0f950068-f8f7-424a-9da8-978105d1e367"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 02:12:15.212923 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.212892 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f950068-f8f7-424a-9da8-978105d1e367-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "0f950068-f8f7-424a-9da8-978105d1e367" (UID: "0f950068-f8f7-424a-9da8-978105d1e367"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 02:12:15.214527 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.214499 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f950068-f8f7-424a-9da8-978105d1e367-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "0f950068-f8f7-424a-9da8-978105d1e367" (UID: "0f950068-f8f7-424a-9da8-978105d1e367"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 02:12:15.220629 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.220610 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f950068-f8f7-424a-9da8-978105d1e367-web-config" (OuterVolumeSpecName: "web-config") pod "0f950068-f8f7-424a-9da8-978105d1e367" (UID: "0f950068-f8f7-424a-9da8-978105d1e367"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 02:12:15.306465 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.306441 2574 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0f950068-f8f7-424a-9da8-978105d1e367-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-141-37.ec2.internal\" DevicePath \"\"" Jul 21 02:12:15.306465 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.306463 2574 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f950068-f8f7-424a-9da8-978105d1e367-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-141-37.ec2.internal\" DevicePath \"\"" Jul 21 02:12:15.306595 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.306472 2574 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0f950068-f8f7-424a-9da8-978105d1e367-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-141-37.ec2.internal\" DevicePath \"\"" Jul 21 02:12:15.306595 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.306482 2574 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0f950068-f8f7-424a-9da8-978105d1e367-tls-assets\") on node \"ip-10-0-141-37.ec2.internal\" DevicePath \"\"" Jul 21 02:12:15.306595 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.306489 2574 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/0f950068-f8f7-424a-9da8-978105d1e367-config\") on node \"ip-10-0-141-37.ec2.internal\" DevicePath \"\"" Jul 21 02:12:15.306595 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.306497 2574 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0f950068-f8f7-424a-9da8-978105d1e367-prometheus-k8s-db\") on node \"ip-10-0-141-37.ec2.internal\" DevicePath \"\"" Jul 21 02:12:15.306595 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.306526 2574 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0f950068-f8f7-424a-9da8-978105d1e367-web-config\") on node \"ip-10-0-141-37.ec2.internal\" DevicePath \"\"" Jul 21 02:12:15.306595 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.306535 2574 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0f950068-f8f7-424a-9da8-978105d1e367-secret-prometheus-k8s-tls\") on node \"ip-10-0-141-37.ec2.internal\" DevicePath \"\"" Jul 21 02:12:15.306595 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.306543 2574 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0f950068-f8f7-424a-9da8-978105d1e367-secret-grpc-tls\") on node \"ip-10-0-141-37.ec2.internal\" DevicePath \"\"" Jul 21 02:12:15.306595 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.306551 2574 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0f950068-f8f7-424a-9da8-978105d1e367-thanos-prometheus-http-client-file\") on node \"ip-10-0-141-37.ec2.internal\" DevicePath \"\"" Jul 21 02:12:15.306595 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.306558 2574 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0f950068-f8f7-424a-9da8-978105d1e367-secret-kube-rbac-proxy\") on node \"ip-10-0-141-37.ec2.internal\" DevicePath \"\"" Jul 21 02:12:15.306595 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.306568 2574 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0f950068-f8f7-424a-9da8-978105d1e367-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-141-37.ec2.internal\" DevicePath \"\"" Jul 21 02:12:15.306595 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.306577 2574 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0f950068-f8f7-424a-9da8-978105d1e367-secret-metrics-client-certs\") on node \"ip-10-0-141-37.ec2.internal\" DevicePath \"\"" Jul 21 02:12:15.478168 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.478085 2574 generic.go:356] "Generic (PLEG): container finished" podID="0f950068-f8f7-424a-9da8-978105d1e367" containerID="af67b4592f00ea2fbf368f75b5606032c0f916e0c97e224ba832e6a380abe3eb" exitCode=0 Jul 21 02:12:15.478168 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.478108 2574 generic.go:356] "Generic (PLEG): container finished" podID="0f950068-f8f7-424a-9da8-978105d1e367" containerID="35ad05add3e363bd4c278dcac693fb8126f149323fcffe2c9069103294f89949" exitCode=0 Jul 21 02:12:15.478168 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.478117 2574 generic.go:356] "Generic (PLEG): container finished" podID="0f950068-f8f7-424a-9da8-978105d1e367" containerID="8751472c673329a5da1c64f6f85f32fb53b08fe66b5fd3c61bb2ac08330b7551" exitCode=0 Jul 21 02:12:15.478168 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.478123 2574 generic.go:356] "Generic (PLEG): container finished" podID="0f950068-f8f7-424a-9da8-978105d1e367" containerID="db57ce3129a039fee6f6bf0b9ed12d22e09a1d89dbf96c265022f8c11b7e440b" exitCode=0 Jul 21 02:12:15.478168 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.478128 2574 generic.go:356] "Generic (PLEG): container finished" podID="0f950068-f8f7-424a-9da8-978105d1e367" containerID="70408f1726561f0969380425235ac6cffca00e3068b14ec98bc43b706867db78" exitCode=0 Jul 21 02:12:15.478168 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.478133 2574 generic.go:356] "Generic (PLEG): container finished" podID="0f950068-f8f7-424a-9da8-978105d1e367" containerID="c3908c1ed6bcd63be65ba6d29732376ff55032b44088afdc4a82c668c6b08f3b" exitCode=0 Jul 21 02:12:15.478168 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.478155 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f950068-f8f7-424a-9da8-978105d1e367","Type":"ContainerDied","Data":"af67b4592f00ea2fbf368f75b5606032c0f916e0c97e224ba832e6a380abe3eb"} Jul 21 02:12:15.478514 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.478165 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:12:15.478514 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.478285 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f950068-f8f7-424a-9da8-978105d1e367","Type":"ContainerDied","Data":"35ad05add3e363bd4c278dcac693fb8126f149323fcffe2c9069103294f89949"} Jul 21 02:12:15.478514 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.478331 2574 scope.go:117] "RemoveContainer" containerID="af67b4592f00ea2fbf368f75b5606032c0f916e0c97e224ba832e6a380abe3eb" Jul 21 02:12:15.478514 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.478367 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f950068-f8f7-424a-9da8-978105d1e367","Type":"ContainerDied","Data":"8751472c673329a5da1c64f6f85f32fb53b08fe66b5fd3c61bb2ac08330b7551"} Jul 21 02:12:15.478514 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.478408 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f950068-f8f7-424a-9da8-978105d1e367","Type":"ContainerDied","Data":"db57ce3129a039fee6f6bf0b9ed12d22e09a1d89dbf96c265022f8c11b7e440b"} Jul 21 02:12:15.478514 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.478450 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f950068-f8f7-424a-9da8-978105d1e367","Type":"ContainerDied","Data":"70408f1726561f0969380425235ac6cffca00e3068b14ec98bc43b706867db78"} Jul 21 02:12:15.478514 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.478492 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f950068-f8f7-424a-9da8-978105d1e367","Type":"ContainerDied","Data":"c3908c1ed6bcd63be65ba6d29732376ff55032b44088afdc4a82c668c6b08f3b"} Jul 21 02:12:15.478749 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.478533 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f950068-f8f7-424a-9da8-978105d1e367","Type":"ContainerDied","Data":"0985646389f21d814b66741e8fe36de7eb2d13b5bcb0366a0c29a0fce2b1d2f3"} Jul 21 02:12:15.486325 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.486228 2574 scope.go:117] "RemoveContainer" containerID="35ad05add3e363bd4c278dcac693fb8126f149323fcffe2c9069103294f89949" Jul 21 02:12:15.493217 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.493203 2574 scope.go:117] "RemoveContainer" containerID="8751472c673329a5da1c64f6f85f32fb53b08fe66b5fd3c61bb2ac08330b7551" Jul 21 02:12:15.499387 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.499369 2574 scope.go:117] "RemoveContainer" containerID="db57ce3129a039fee6f6bf0b9ed12d22e09a1d89dbf96c265022f8c11b7e440b" Jul 21 02:12:15.504460 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.504439 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 21 02:12:15.506201 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.506171 2574 scope.go:117] "RemoveContainer" containerID="70408f1726561f0969380425235ac6cffca00e3068b14ec98bc43b706867db78" Jul 21 02:12:15.509067 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.509048 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 21 02:12:15.513053 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.513038 2574 scope.go:117] "RemoveContainer" containerID="c3908c1ed6bcd63be65ba6d29732376ff55032b44088afdc4a82c668c6b08f3b" Jul 21 02:12:15.519562 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.519548 2574 scope.go:117] "RemoveContainer" containerID="9edf3aea3263a0d1ac1401f349f9d2285dd6c6f5f2f3e092f77bbd4affbe52ba" Jul 21 02:12:15.525793 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.525780 2574 scope.go:117] "RemoveContainer" containerID="af67b4592f00ea2fbf368f75b5606032c0f916e0c97e224ba832e6a380abe3eb" Jul 21 02:12:15.526003 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:12:15.525988 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af67b4592f00ea2fbf368f75b5606032c0f916e0c97e224ba832e6a380abe3eb\": container with ID starting with af67b4592f00ea2fbf368f75b5606032c0f916e0c97e224ba832e6a380abe3eb not found: ID does not exist" containerID="af67b4592f00ea2fbf368f75b5606032c0f916e0c97e224ba832e6a380abe3eb" Jul 21 02:12:15.526051 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.526012 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af67b4592f00ea2fbf368f75b5606032c0f916e0c97e224ba832e6a380abe3eb"} err="failed to get container status \"af67b4592f00ea2fbf368f75b5606032c0f916e0c97e224ba832e6a380abe3eb\": rpc error: code = NotFound desc = could not find container \"af67b4592f00ea2fbf368f75b5606032c0f916e0c97e224ba832e6a380abe3eb\": container with ID starting with af67b4592f00ea2fbf368f75b5606032c0f916e0c97e224ba832e6a380abe3eb not found: ID does not exist" Jul 21 02:12:15.526051 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.526028 2574 scope.go:117] "RemoveContainer" containerID="35ad05add3e363bd4c278dcac693fb8126f149323fcffe2c9069103294f89949" Jul 21 02:12:15.526228 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:12:15.526215 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35ad05add3e363bd4c278dcac693fb8126f149323fcffe2c9069103294f89949\": container with ID starting with 35ad05add3e363bd4c278dcac693fb8126f149323fcffe2c9069103294f89949 not found: ID does not exist" containerID="35ad05add3e363bd4c278dcac693fb8126f149323fcffe2c9069103294f89949" Jul 21 02:12:15.526295 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.526232 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35ad05add3e363bd4c278dcac693fb8126f149323fcffe2c9069103294f89949"} err="failed to get container status \"35ad05add3e363bd4c278dcac693fb8126f149323fcffe2c9069103294f89949\": rpc error: code = NotFound desc = could not find container \"35ad05add3e363bd4c278dcac693fb8126f149323fcffe2c9069103294f89949\": container with ID starting with 35ad05add3e363bd4c278dcac693fb8126f149323fcffe2c9069103294f89949 not found: ID does not exist" Jul 21 02:12:15.526295 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.526245 2574 scope.go:117] "RemoveContainer" containerID="8751472c673329a5da1c64f6f85f32fb53b08fe66b5fd3c61bb2ac08330b7551" Jul 21 02:12:15.526483 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:12:15.526468 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8751472c673329a5da1c64f6f85f32fb53b08fe66b5fd3c61bb2ac08330b7551\": container with ID starting with 8751472c673329a5da1c64f6f85f32fb53b08fe66b5fd3c61bb2ac08330b7551 not found: ID does not exist" containerID="8751472c673329a5da1c64f6f85f32fb53b08fe66b5fd3c61bb2ac08330b7551" Jul 21 02:12:15.526528 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.526486 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8751472c673329a5da1c64f6f85f32fb53b08fe66b5fd3c61bb2ac08330b7551"} err="failed to get container status \"8751472c673329a5da1c64f6f85f32fb53b08fe66b5fd3c61bb2ac08330b7551\": rpc error: code = NotFound desc = could not find container \"8751472c673329a5da1c64f6f85f32fb53b08fe66b5fd3c61bb2ac08330b7551\": container with ID starting with 8751472c673329a5da1c64f6f85f32fb53b08fe66b5fd3c61bb2ac08330b7551 not found: ID does not exist" Jul 21 02:12:15.526528 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.526497 2574 scope.go:117] "RemoveContainer" containerID="db57ce3129a039fee6f6bf0b9ed12d22e09a1d89dbf96c265022f8c11b7e440b" Jul 21 02:12:15.526707 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:12:15.526692 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db57ce3129a039fee6f6bf0b9ed12d22e09a1d89dbf96c265022f8c11b7e440b\": container with ID starting with db57ce3129a039fee6f6bf0b9ed12d22e09a1d89dbf96c265022f8c11b7e440b not found: ID does not exist" containerID="db57ce3129a039fee6f6bf0b9ed12d22e09a1d89dbf96c265022f8c11b7e440b" Jul 21 02:12:15.526743 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.526711 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db57ce3129a039fee6f6bf0b9ed12d22e09a1d89dbf96c265022f8c11b7e440b"} err="failed to get container status \"db57ce3129a039fee6f6bf0b9ed12d22e09a1d89dbf96c265022f8c11b7e440b\": rpc error: code = NotFound desc = could not find container \"db57ce3129a039fee6f6bf0b9ed12d22e09a1d89dbf96c265022f8c11b7e440b\": container with ID starting with db57ce3129a039fee6f6bf0b9ed12d22e09a1d89dbf96c265022f8c11b7e440b not found: ID does not exist" Jul 21 02:12:15.526743 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.526722 2574 scope.go:117] "RemoveContainer" containerID="70408f1726561f0969380425235ac6cffca00e3068b14ec98bc43b706867db78" Jul 21 02:12:15.526936 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:12:15.526923 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70408f1726561f0969380425235ac6cffca00e3068b14ec98bc43b706867db78\": container with ID starting with 70408f1726561f0969380425235ac6cffca00e3068b14ec98bc43b706867db78 not found: ID does not exist" containerID="70408f1726561f0969380425235ac6cffca00e3068b14ec98bc43b706867db78" Jul 21 02:12:15.526980 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.526940 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70408f1726561f0969380425235ac6cffca00e3068b14ec98bc43b706867db78"} err="failed to get container status \"70408f1726561f0969380425235ac6cffca00e3068b14ec98bc43b706867db78\": rpc error: code = NotFound desc = could not find container \"70408f1726561f0969380425235ac6cffca00e3068b14ec98bc43b706867db78\": container with ID starting with 70408f1726561f0969380425235ac6cffca00e3068b14ec98bc43b706867db78 not found: ID does not exist" Jul 21 02:12:15.526980 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.526951 2574 scope.go:117] "RemoveContainer" containerID="c3908c1ed6bcd63be65ba6d29732376ff55032b44088afdc4a82c668c6b08f3b" Jul 21 02:12:15.527141 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:12:15.527127 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3908c1ed6bcd63be65ba6d29732376ff55032b44088afdc4a82c668c6b08f3b\": container with ID starting with c3908c1ed6bcd63be65ba6d29732376ff55032b44088afdc4a82c668c6b08f3b not found: ID does not exist" containerID="c3908c1ed6bcd63be65ba6d29732376ff55032b44088afdc4a82c668c6b08f3b" Jul 21 02:12:15.527180 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.527144 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3908c1ed6bcd63be65ba6d29732376ff55032b44088afdc4a82c668c6b08f3b"} err="failed to get container status \"c3908c1ed6bcd63be65ba6d29732376ff55032b44088afdc4a82c668c6b08f3b\": rpc error: code = NotFound desc = could not find container \"c3908c1ed6bcd63be65ba6d29732376ff55032b44088afdc4a82c668c6b08f3b\": container with ID starting with c3908c1ed6bcd63be65ba6d29732376ff55032b44088afdc4a82c668c6b08f3b not found: ID does not exist" Jul 21 02:12:15.527180 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.527156 2574 scope.go:117] "RemoveContainer" containerID="9edf3aea3263a0d1ac1401f349f9d2285dd6c6f5f2f3e092f77bbd4affbe52ba" Jul 21 02:12:15.527377 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:12:15.527362 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9edf3aea3263a0d1ac1401f349f9d2285dd6c6f5f2f3e092f77bbd4affbe52ba\": container with ID starting with 9edf3aea3263a0d1ac1401f349f9d2285dd6c6f5f2f3e092f77bbd4affbe52ba not found: ID does not exist" containerID="9edf3aea3263a0d1ac1401f349f9d2285dd6c6f5f2f3e092f77bbd4affbe52ba" Jul 21 02:12:15.527423 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.527379 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9edf3aea3263a0d1ac1401f349f9d2285dd6c6f5f2f3e092f77bbd4affbe52ba"} err="failed to get container status \"9edf3aea3263a0d1ac1401f349f9d2285dd6c6f5f2f3e092f77bbd4affbe52ba\": rpc error: code = NotFound desc = could not find container \"9edf3aea3263a0d1ac1401f349f9d2285dd6c6f5f2f3e092f77bbd4affbe52ba\": container with ID starting with 9edf3aea3263a0d1ac1401f349f9d2285dd6c6f5f2f3e092f77bbd4affbe52ba not found: ID does not exist" Jul 21 02:12:15.527423 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.527392 2574 scope.go:117] "RemoveContainer" containerID="af67b4592f00ea2fbf368f75b5606032c0f916e0c97e224ba832e6a380abe3eb" Jul 21 02:12:15.527582 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.527568 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af67b4592f00ea2fbf368f75b5606032c0f916e0c97e224ba832e6a380abe3eb"} err="failed to get container status \"af67b4592f00ea2fbf368f75b5606032c0f916e0c97e224ba832e6a380abe3eb\": rpc error: code = NotFound desc = could not find container \"af67b4592f00ea2fbf368f75b5606032c0f916e0c97e224ba832e6a380abe3eb\": container with ID starting with af67b4592f00ea2fbf368f75b5606032c0f916e0c97e224ba832e6a380abe3eb not found: ID does not exist" Jul 21 02:12:15.527623 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.527582 2574 scope.go:117] "RemoveContainer" containerID="35ad05add3e363bd4c278dcac693fb8126f149323fcffe2c9069103294f89949" Jul 21 02:12:15.527828 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.527809 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35ad05add3e363bd4c278dcac693fb8126f149323fcffe2c9069103294f89949"} err="failed to get container status \"35ad05add3e363bd4c278dcac693fb8126f149323fcffe2c9069103294f89949\": rpc error: code = NotFound desc = could not find container \"35ad05add3e363bd4c278dcac693fb8126f149323fcffe2c9069103294f89949\": container with ID starting with 35ad05add3e363bd4c278dcac693fb8126f149323fcffe2c9069103294f89949 not found: ID does not exist" Jul 21 02:12:15.527878 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.527829 2574 scope.go:117] "RemoveContainer" containerID="8751472c673329a5da1c64f6f85f32fb53b08fe66b5fd3c61bb2ac08330b7551" Jul 21 02:12:15.528058 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.528043 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8751472c673329a5da1c64f6f85f32fb53b08fe66b5fd3c61bb2ac08330b7551"} err="failed to get container status \"8751472c673329a5da1c64f6f85f32fb53b08fe66b5fd3c61bb2ac08330b7551\": rpc error: code = NotFound desc = could not find container \"8751472c673329a5da1c64f6f85f32fb53b08fe66b5fd3c61bb2ac08330b7551\": container with ID starting with 8751472c673329a5da1c64f6f85f32fb53b08fe66b5fd3c61bb2ac08330b7551 not found: ID does not exist" Jul 21 02:12:15.528106 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.528059 2574 scope.go:117] "RemoveContainer" containerID="db57ce3129a039fee6f6bf0b9ed12d22e09a1d89dbf96c265022f8c11b7e440b" Jul 21 02:12:15.528294 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.528244 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db57ce3129a039fee6f6bf0b9ed12d22e09a1d89dbf96c265022f8c11b7e440b"} err="failed to get container status \"db57ce3129a039fee6f6bf0b9ed12d22e09a1d89dbf96c265022f8c11b7e440b\": rpc error: code = NotFound desc = could not find container \"db57ce3129a039fee6f6bf0b9ed12d22e09a1d89dbf96c265022f8c11b7e440b\": container with ID starting with db57ce3129a039fee6f6bf0b9ed12d22e09a1d89dbf96c265022f8c11b7e440b not found: ID does not exist" Jul 21 02:12:15.528294 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.528281 2574 scope.go:117] "RemoveContainer" containerID="70408f1726561f0969380425235ac6cffca00e3068b14ec98bc43b706867db78" Jul 21 02:12:15.528490 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.528472 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70408f1726561f0969380425235ac6cffca00e3068b14ec98bc43b706867db78"} err="failed to get container status \"70408f1726561f0969380425235ac6cffca00e3068b14ec98bc43b706867db78\": rpc error: code = NotFound desc = could not find container \"70408f1726561f0969380425235ac6cffca00e3068b14ec98bc43b706867db78\": container with ID starting with 70408f1726561f0969380425235ac6cffca00e3068b14ec98bc43b706867db78 not found: ID does not exist" Jul 21 02:12:15.528545 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.528493 2574 scope.go:117] "RemoveContainer" containerID="c3908c1ed6bcd63be65ba6d29732376ff55032b44088afdc4a82c668c6b08f3b" Jul 21 02:12:15.528696 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.528681 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3908c1ed6bcd63be65ba6d29732376ff55032b44088afdc4a82c668c6b08f3b"} err="failed to get container status \"c3908c1ed6bcd63be65ba6d29732376ff55032b44088afdc4a82c668c6b08f3b\": rpc error: code = NotFound desc = could not find container \"c3908c1ed6bcd63be65ba6d29732376ff55032b44088afdc4a82c668c6b08f3b\": container with ID starting with c3908c1ed6bcd63be65ba6d29732376ff55032b44088afdc4a82c668c6b08f3b not found: ID does not exist" Jul 21 02:12:15.528736 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.528697 2574 scope.go:117] "RemoveContainer" containerID="9edf3aea3263a0d1ac1401f349f9d2285dd6c6f5f2f3e092f77bbd4affbe52ba" Jul 21 02:12:15.528930 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.528908 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9edf3aea3263a0d1ac1401f349f9d2285dd6c6f5f2f3e092f77bbd4affbe52ba"} err="failed to get container status \"9edf3aea3263a0d1ac1401f349f9d2285dd6c6f5f2f3e092f77bbd4affbe52ba\": rpc error: code = NotFound desc = could not find container \"9edf3aea3263a0d1ac1401f349f9d2285dd6c6f5f2f3e092f77bbd4affbe52ba\": container with ID starting with 9edf3aea3263a0d1ac1401f349f9d2285dd6c6f5f2f3e092f77bbd4affbe52ba not found: ID does not exist" Jul 21 02:12:15.529000 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.528932 2574 scope.go:117] "RemoveContainer" containerID="af67b4592f00ea2fbf368f75b5606032c0f916e0c97e224ba832e6a380abe3eb" Jul 21 02:12:15.529180 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.529164 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af67b4592f00ea2fbf368f75b5606032c0f916e0c97e224ba832e6a380abe3eb"} err="failed to get container status \"af67b4592f00ea2fbf368f75b5606032c0f916e0c97e224ba832e6a380abe3eb\": rpc error: code = NotFound desc = could not find container \"af67b4592f00ea2fbf368f75b5606032c0f916e0c97e224ba832e6a380abe3eb\": container with ID starting with af67b4592f00ea2fbf368f75b5606032c0f916e0c97e224ba832e6a380abe3eb not found: ID does not exist" Jul 21 02:12:15.529228 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.529181 2574 scope.go:117] "RemoveContainer" containerID="35ad05add3e363bd4c278dcac693fb8126f149323fcffe2c9069103294f89949" Jul 21 02:12:15.529372 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.529356 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35ad05add3e363bd4c278dcac693fb8126f149323fcffe2c9069103294f89949"} err="failed to get container status \"35ad05add3e363bd4c278dcac693fb8126f149323fcffe2c9069103294f89949\": rpc error: code = NotFound desc = could not find container \"35ad05add3e363bd4c278dcac693fb8126f149323fcffe2c9069103294f89949\": container with ID starting with 35ad05add3e363bd4c278dcac693fb8126f149323fcffe2c9069103294f89949 not found: ID does not exist" Jul 21 02:12:15.529435 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.529373 2574 scope.go:117] "RemoveContainer" containerID="8751472c673329a5da1c64f6f85f32fb53b08fe66b5fd3c61bb2ac08330b7551" Jul 21 02:12:15.529566 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.529551 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8751472c673329a5da1c64f6f85f32fb53b08fe66b5fd3c61bb2ac08330b7551"} err="failed to get container status \"8751472c673329a5da1c64f6f85f32fb53b08fe66b5fd3c61bb2ac08330b7551\": rpc error: code = NotFound desc = could not find container \"8751472c673329a5da1c64f6f85f32fb53b08fe66b5fd3c61bb2ac08330b7551\": container with ID starting with 8751472c673329a5da1c64f6f85f32fb53b08fe66b5fd3c61bb2ac08330b7551 not found: ID does not exist" Jul 21 02:12:15.529566 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.529566 2574 scope.go:117] "RemoveContainer" containerID="db57ce3129a039fee6f6bf0b9ed12d22e09a1d89dbf96c265022f8c11b7e440b" Jul 21 02:12:15.529773 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.529757 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db57ce3129a039fee6f6bf0b9ed12d22e09a1d89dbf96c265022f8c11b7e440b"} err="failed to get container status \"db57ce3129a039fee6f6bf0b9ed12d22e09a1d89dbf96c265022f8c11b7e440b\": rpc error: code = NotFound desc = could not find container \"db57ce3129a039fee6f6bf0b9ed12d22e09a1d89dbf96c265022f8c11b7e440b\": container with ID starting with db57ce3129a039fee6f6bf0b9ed12d22e09a1d89dbf96c265022f8c11b7e440b not found: ID does not exist" Jul 21 02:12:15.529773 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.529772 2574 scope.go:117] "RemoveContainer" containerID="70408f1726561f0969380425235ac6cffca00e3068b14ec98bc43b706867db78" Jul 21 02:12:15.529947 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.529933 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70408f1726561f0969380425235ac6cffca00e3068b14ec98bc43b706867db78"} err="failed to get container status \"70408f1726561f0969380425235ac6cffca00e3068b14ec98bc43b706867db78\": rpc error: code = NotFound desc = could not find container \"70408f1726561f0969380425235ac6cffca00e3068b14ec98bc43b706867db78\": container with ID starting with 70408f1726561f0969380425235ac6cffca00e3068b14ec98bc43b706867db78 not found: ID does not exist" Jul 21 02:12:15.529991 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.529948 2574 scope.go:117] "RemoveContainer" containerID="c3908c1ed6bcd63be65ba6d29732376ff55032b44088afdc4a82c668c6b08f3b" Jul 21 02:12:15.530137 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.530122 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3908c1ed6bcd63be65ba6d29732376ff55032b44088afdc4a82c668c6b08f3b"} err="failed to get container status \"c3908c1ed6bcd63be65ba6d29732376ff55032b44088afdc4a82c668c6b08f3b\": rpc error: code = NotFound desc = could not find container \"c3908c1ed6bcd63be65ba6d29732376ff55032b44088afdc4a82c668c6b08f3b\": container with ID starting with c3908c1ed6bcd63be65ba6d29732376ff55032b44088afdc4a82c668c6b08f3b not found: ID does not exist" Jul 21 02:12:15.530137 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.530137 2574 scope.go:117] "RemoveContainer" containerID="9edf3aea3263a0d1ac1401f349f9d2285dd6c6f5f2f3e092f77bbd4affbe52ba" Jul 21 02:12:15.530352 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.530333 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9edf3aea3263a0d1ac1401f349f9d2285dd6c6f5f2f3e092f77bbd4affbe52ba"} err="failed to get container status \"9edf3aea3263a0d1ac1401f349f9d2285dd6c6f5f2f3e092f77bbd4affbe52ba\": rpc error: code = NotFound desc = could not find container \"9edf3aea3263a0d1ac1401f349f9d2285dd6c6f5f2f3e092f77bbd4affbe52ba\": container with ID starting with 9edf3aea3263a0d1ac1401f349f9d2285dd6c6f5f2f3e092f77bbd4affbe52ba not found: ID does not exist" Jul 21 02:12:15.530390 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.530352 2574 scope.go:117] "RemoveContainer" containerID="af67b4592f00ea2fbf368f75b5606032c0f916e0c97e224ba832e6a380abe3eb" Jul 21 02:12:15.530549 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.530526 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af67b4592f00ea2fbf368f75b5606032c0f916e0c97e224ba832e6a380abe3eb"} err="failed to get container status \"af67b4592f00ea2fbf368f75b5606032c0f916e0c97e224ba832e6a380abe3eb\": rpc error: code = NotFound desc = could not find container \"af67b4592f00ea2fbf368f75b5606032c0f916e0c97e224ba832e6a380abe3eb\": container with ID starting with af67b4592f00ea2fbf368f75b5606032c0f916e0c97e224ba832e6a380abe3eb not found: ID does not exist" Jul 21 02:12:15.530549 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.530548 2574 scope.go:117] "RemoveContainer" containerID="35ad05add3e363bd4c278dcac693fb8126f149323fcffe2c9069103294f89949" Jul 21 02:12:15.530716 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.530702 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35ad05add3e363bd4c278dcac693fb8126f149323fcffe2c9069103294f89949"} err="failed to get container status \"35ad05add3e363bd4c278dcac693fb8126f149323fcffe2c9069103294f89949\": rpc error: code = NotFound desc = could not find container \"35ad05add3e363bd4c278dcac693fb8126f149323fcffe2c9069103294f89949\": container with ID starting with 35ad05add3e363bd4c278dcac693fb8126f149323fcffe2c9069103294f89949 not found: ID does not exist" Jul 21 02:12:15.530716 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.530714 2574 scope.go:117] "RemoveContainer" containerID="8751472c673329a5da1c64f6f85f32fb53b08fe66b5fd3c61bb2ac08330b7551" Jul 21 02:12:15.530953 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.530927 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8751472c673329a5da1c64f6f85f32fb53b08fe66b5fd3c61bb2ac08330b7551"} err="failed to get container status \"8751472c673329a5da1c64f6f85f32fb53b08fe66b5fd3c61bb2ac08330b7551\": rpc error: code = NotFound desc = could not find container \"8751472c673329a5da1c64f6f85f32fb53b08fe66b5fd3c61bb2ac08330b7551\": container with ID starting with 8751472c673329a5da1c64f6f85f32fb53b08fe66b5fd3c61bb2ac08330b7551 not found: ID does not exist" Jul 21 02:12:15.530953 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.530947 2574 scope.go:117] "RemoveContainer" containerID="db57ce3129a039fee6f6bf0b9ed12d22e09a1d89dbf96c265022f8c11b7e440b" Jul 21 02:12:15.531164 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.531148 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db57ce3129a039fee6f6bf0b9ed12d22e09a1d89dbf96c265022f8c11b7e440b"} err="failed to get container status \"db57ce3129a039fee6f6bf0b9ed12d22e09a1d89dbf96c265022f8c11b7e440b\": rpc error: code = NotFound desc = could not find container \"db57ce3129a039fee6f6bf0b9ed12d22e09a1d89dbf96c265022f8c11b7e440b\": container with ID starting with db57ce3129a039fee6f6bf0b9ed12d22e09a1d89dbf96c265022f8c11b7e440b not found: ID does not exist" Jul 21 02:12:15.531229 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.531165 2574 scope.go:117] "RemoveContainer" containerID="70408f1726561f0969380425235ac6cffca00e3068b14ec98bc43b706867db78" Jul 21 02:12:15.531390 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.531375 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70408f1726561f0969380425235ac6cffca00e3068b14ec98bc43b706867db78"} err="failed to get container status \"70408f1726561f0969380425235ac6cffca00e3068b14ec98bc43b706867db78\": rpc error: code = NotFound desc = could not find container \"70408f1726561f0969380425235ac6cffca00e3068b14ec98bc43b706867db78\": container with ID starting with 70408f1726561f0969380425235ac6cffca00e3068b14ec98bc43b706867db78 not found: ID does not exist" Jul 21 02:12:15.531430 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.531390 2574 scope.go:117] "RemoveContainer" containerID="c3908c1ed6bcd63be65ba6d29732376ff55032b44088afdc4a82c668c6b08f3b" Jul 21 02:12:15.531604 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.531589 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3908c1ed6bcd63be65ba6d29732376ff55032b44088afdc4a82c668c6b08f3b"} err="failed to get container status \"c3908c1ed6bcd63be65ba6d29732376ff55032b44088afdc4a82c668c6b08f3b\": rpc error: code = NotFound desc = could not find container \"c3908c1ed6bcd63be65ba6d29732376ff55032b44088afdc4a82c668c6b08f3b\": container with ID starting with c3908c1ed6bcd63be65ba6d29732376ff55032b44088afdc4a82c668c6b08f3b not found: ID does not exist" Jul 21 02:12:15.531654 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.531604 2574 scope.go:117] "RemoveContainer" containerID="9edf3aea3263a0d1ac1401f349f9d2285dd6c6f5f2f3e092f77bbd4affbe52ba" Jul 21 02:12:15.531801 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.531785 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9edf3aea3263a0d1ac1401f349f9d2285dd6c6f5f2f3e092f77bbd4affbe52ba"} err="failed to get container status \"9edf3aea3263a0d1ac1401f349f9d2285dd6c6f5f2f3e092f77bbd4affbe52ba\": rpc error: code = NotFound desc = could not find container \"9edf3aea3263a0d1ac1401f349f9d2285dd6c6f5f2f3e092f77bbd4affbe52ba\": container with ID starting with 9edf3aea3263a0d1ac1401f349f9d2285dd6c6f5f2f3e092f77bbd4affbe52ba not found: ID does not exist" Jul 21 02:12:15.531837 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.531801 2574 scope.go:117] "RemoveContainer" containerID="af67b4592f00ea2fbf368f75b5606032c0f916e0c97e224ba832e6a380abe3eb" Jul 21 02:12:15.532028 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.532012 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af67b4592f00ea2fbf368f75b5606032c0f916e0c97e224ba832e6a380abe3eb"} err="failed to get container status \"af67b4592f00ea2fbf368f75b5606032c0f916e0c97e224ba832e6a380abe3eb\": rpc error: code = NotFound desc = could not find container \"af67b4592f00ea2fbf368f75b5606032c0f916e0c97e224ba832e6a380abe3eb\": container with ID starting with af67b4592f00ea2fbf368f75b5606032c0f916e0c97e224ba832e6a380abe3eb not found: ID does not exist" Jul 21 02:12:15.532075 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.532028 2574 scope.go:117] "RemoveContainer" containerID="35ad05add3e363bd4c278dcac693fb8126f149323fcffe2c9069103294f89949" Jul 21 02:12:15.532225 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.532207 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35ad05add3e363bd4c278dcac693fb8126f149323fcffe2c9069103294f89949"} err="failed to get container status \"35ad05add3e363bd4c278dcac693fb8126f149323fcffe2c9069103294f89949\": rpc error: code = NotFound desc = could not find container \"35ad05add3e363bd4c278dcac693fb8126f149323fcffe2c9069103294f89949\": container with ID starting with 35ad05add3e363bd4c278dcac693fb8126f149323fcffe2c9069103294f89949 not found: ID does not exist" Jul 21 02:12:15.532280 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.532226 2574 scope.go:117] "RemoveContainer" containerID="8751472c673329a5da1c64f6f85f32fb53b08fe66b5fd3c61bb2ac08330b7551" Jul 21 02:12:15.532509 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.532491 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8751472c673329a5da1c64f6f85f32fb53b08fe66b5fd3c61bb2ac08330b7551"} err="failed to get container status \"8751472c673329a5da1c64f6f85f32fb53b08fe66b5fd3c61bb2ac08330b7551\": rpc error: code = NotFound desc = could not find container \"8751472c673329a5da1c64f6f85f32fb53b08fe66b5fd3c61bb2ac08330b7551\": container with ID starting with 8751472c673329a5da1c64f6f85f32fb53b08fe66b5fd3c61bb2ac08330b7551 not found: ID does not exist" Jul 21 02:12:15.532509 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.532510 2574 scope.go:117] "RemoveContainer" containerID="db57ce3129a039fee6f6bf0b9ed12d22e09a1d89dbf96c265022f8c11b7e440b" Jul 21 02:12:15.532844 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.532789 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db57ce3129a039fee6f6bf0b9ed12d22e09a1d89dbf96c265022f8c11b7e440b"} err="failed to get container status \"db57ce3129a039fee6f6bf0b9ed12d22e09a1d89dbf96c265022f8c11b7e440b\": rpc error: code = NotFound desc = could not find container \"db57ce3129a039fee6f6bf0b9ed12d22e09a1d89dbf96c265022f8c11b7e440b\": container with ID starting with db57ce3129a039fee6f6bf0b9ed12d22e09a1d89dbf96c265022f8c11b7e440b not found: ID does not exist" Jul 21 02:12:15.532844 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.532830 2574 scope.go:117] "RemoveContainer" containerID="70408f1726561f0969380425235ac6cffca00e3068b14ec98bc43b706867db78" Jul 21 02:12:15.533172 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.533131 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70408f1726561f0969380425235ac6cffca00e3068b14ec98bc43b706867db78"} err="failed to get container status \"70408f1726561f0969380425235ac6cffca00e3068b14ec98bc43b706867db78\": rpc error: code = NotFound desc = could not find container \"70408f1726561f0969380425235ac6cffca00e3068b14ec98bc43b706867db78\": container with ID starting with 70408f1726561f0969380425235ac6cffca00e3068b14ec98bc43b706867db78 not found: ID does not exist" Jul 21 02:12:15.533253 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.533179 2574 scope.go:117] "RemoveContainer" containerID="c3908c1ed6bcd63be65ba6d29732376ff55032b44088afdc4a82c668c6b08f3b" Jul 21 02:12:15.533638 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.533607 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3908c1ed6bcd63be65ba6d29732376ff55032b44088afdc4a82c668c6b08f3b"} err="failed to get container status \"c3908c1ed6bcd63be65ba6d29732376ff55032b44088afdc4a82c668c6b08f3b\": rpc error: code = NotFound desc = could not find container \"c3908c1ed6bcd63be65ba6d29732376ff55032b44088afdc4a82c668c6b08f3b\": container with ID starting with c3908c1ed6bcd63be65ba6d29732376ff55032b44088afdc4a82c668c6b08f3b not found: ID does not exist" Jul 21 02:12:15.533718 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.533640 2574 scope.go:117] "RemoveContainer" containerID="9edf3aea3263a0d1ac1401f349f9d2285dd6c6f5f2f3e092f77bbd4affbe52ba" Jul 21 02:12:15.533932 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.533913 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9edf3aea3263a0d1ac1401f349f9d2285dd6c6f5f2f3e092f77bbd4affbe52ba"} err="failed to get container status \"9edf3aea3263a0d1ac1401f349f9d2285dd6c6f5f2f3e092f77bbd4affbe52ba\": rpc error: code = NotFound desc = could not find container \"9edf3aea3263a0d1ac1401f349f9d2285dd6c6f5f2f3e092f77bbd4affbe52ba\": container with ID starting with 9edf3aea3263a0d1ac1401f349f9d2285dd6c6f5f2f3e092f77bbd4affbe52ba not found: ID does not exist" Jul 21 02:12:15.533932 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.533932 2574 scope.go:117] "RemoveContainer" containerID="af67b4592f00ea2fbf368f75b5606032c0f916e0c97e224ba832e6a380abe3eb" Jul 21 02:12:15.534152 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.534135 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af67b4592f00ea2fbf368f75b5606032c0f916e0c97e224ba832e6a380abe3eb"} err="failed to get container status \"af67b4592f00ea2fbf368f75b5606032c0f916e0c97e224ba832e6a380abe3eb\": rpc error: code = NotFound desc = could not find container \"af67b4592f00ea2fbf368f75b5606032c0f916e0c97e224ba832e6a380abe3eb\": container with ID starting with af67b4592f00ea2fbf368f75b5606032c0f916e0c97e224ba832e6a380abe3eb not found: ID does not exist" Jul 21 02:12:15.534201 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.534153 2574 scope.go:117] "RemoveContainer" containerID="35ad05add3e363bd4c278dcac693fb8126f149323fcffe2c9069103294f89949" Jul 21 02:12:15.534411 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.534389 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35ad05add3e363bd4c278dcac693fb8126f149323fcffe2c9069103294f89949"} err="failed to get container status \"35ad05add3e363bd4c278dcac693fb8126f149323fcffe2c9069103294f89949\": rpc error: code = NotFound desc = could not find container \"35ad05add3e363bd4c278dcac693fb8126f149323fcffe2c9069103294f89949\": container with ID starting with 35ad05add3e363bd4c278dcac693fb8126f149323fcffe2c9069103294f89949 not found: ID does not exist" Jul 21 02:12:15.534411 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.534410 2574 scope.go:117] "RemoveContainer" containerID="8751472c673329a5da1c64f6f85f32fb53b08fe66b5fd3c61bb2ac08330b7551" Jul 21 02:12:15.534697 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.534677 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8751472c673329a5da1c64f6f85f32fb53b08fe66b5fd3c61bb2ac08330b7551"} err="failed to get container status \"8751472c673329a5da1c64f6f85f32fb53b08fe66b5fd3c61bb2ac08330b7551\": rpc error: code = NotFound desc = could not find container \"8751472c673329a5da1c64f6f85f32fb53b08fe66b5fd3c61bb2ac08330b7551\": container with ID starting with 8751472c673329a5da1c64f6f85f32fb53b08fe66b5fd3c61bb2ac08330b7551 not found: ID does not exist" Jul 21 02:12:15.534697 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.534697 2574 scope.go:117] "RemoveContainer" containerID="db57ce3129a039fee6f6bf0b9ed12d22e09a1d89dbf96c265022f8c11b7e440b" Jul 21 02:12:15.534916 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.534899 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db57ce3129a039fee6f6bf0b9ed12d22e09a1d89dbf96c265022f8c11b7e440b"} err="failed to get container status \"db57ce3129a039fee6f6bf0b9ed12d22e09a1d89dbf96c265022f8c11b7e440b\": rpc error: code = NotFound desc = could not find container \"db57ce3129a039fee6f6bf0b9ed12d22e09a1d89dbf96c265022f8c11b7e440b\": container with ID starting with db57ce3129a039fee6f6bf0b9ed12d22e09a1d89dbf96c265022f8c11b7e440b not found: ID does not exist" Jul 21 02:12:15.534964 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.534918 2574 scope.go:117] "RemoveContainer" containerID="70408f1726561f0969380425235ac6cffca00e3068b14ec98bc43b706867db78" Jul 21 02:12:15.535147 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.535128 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70408f1726561f0969380425235ac6cffca00e3068b14ec98bc43b706867db78"} err="failed to get container status \"70408f1726561f0969380425235ac6cffca00e3068b14ec98bc43b706867db78\": rpc error: code = NotFound desc = could not find container \"70408f1726561f0969380425235ac6cffca00e3068b14ec98bc43b706867db78\": container with ID starting with 70408f1726561f0969380425235ac6cffca00e3068b14ec98bc43b706867db78 not found: ID does not exist" Jul 21 02:12:15.535147 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.535148 2574 scope.go:117] "RemoveContainer" containerID="c3908c1ed6bcd63be65ba6d29732376ff55032b44088afdc4a82c668c6b08f3b" Jul 21 02:12:15.535294 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.535133 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 21 02:12:15.535412 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.535395 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3908c1ed6bcd63be65ba6d29732376ff55032b44088afdc4a82c668c6b08f3b"} err="failed to get container status \"c3908c1ed6bcd63be65ba6d29732376ff55032b44088afdc4a82c668c6b08f3b\": rpc error: code = NotFound desc = could not find container \"c3908c1ed6bcd63be65ba6d29732376ff55032b44088afdc4a82c668c6b08f3b\": container with ID starting with c3908c1ed6bcd63be65ba6d29732376ff55032b44088afdc4a82c668c6b08f3b not found: ID does not exist" Jul 21 02:12:15.535465 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.535414 2574 scope.go:117] "RemoveContainer" containerID="9edf3aea3263a0d1ac1401f349f9d2285dd6c6f5f2f3e092f77bbd4affbe52ba" Jul 21 02:12:15.535465 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.535449 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2694a3ee-0897-4753-bfee-ff58550df666" containerName="init-config-reloader" Jul 21 02:12:15.535465 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.535460 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2694a3ee-0897-4753-bfee-ff58550df666" containerName="init-config-reloader" Jul 21 02:12:15.535595 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.535470 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2694a3ee-0897-4753-bfee-ff58550df666" containerName="alertmanager" Jul 21 02:12:15.535595 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.535476 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2694a3ee-0897-4753-bfee-ff58550df666" containerName="alertmanager" Jul 21 02:12:15.535595 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.535490 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f950068-f8f7-424a-9da8-978105d1e367" containerName="prometheus" Jul 21 02:12:15.535595 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.535499 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f950068-f8f7-424a-9da8-978105d1e367" containerName="prometheus" Jul 21 02:12:15.535595 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.535505 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f950068-f8f7-424a-9da8-978105d1e367" containerName="config-reloader" Jul 21 02:12:15.535595 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.535510 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f950068-f8f7-424a-9da8-978105d1e367" containerName="config-reloader" Jul 21 02:12:15.535595 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.535515 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f950068-f8f7-424a-9da8-978105d1e367" containerName="kube-rbac-proxy-web" Jul 21 02:12:15.535595 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.535521 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f950068-f8f7-424a-9da8-978105d1e367" containerName="kube-rbac-proxy-web" Jul 21 02:12:15.535595 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.535543 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f950068-f8f7-424a-9da8-978105d1e367" containerName="kube-rbac-proxy-thanos" Jul 21 02:12:15.535595 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.535551 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f950068-f8f7-424a-9da8-978105d1e367" containerName="kube-rbac-proxy-thanos" Jul 21 02:12:15.535595 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.535562 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2694a3ee-0897-4753-bfee-ff58550df666" containerName="config-reloader" Jul 21 02:12:15.535595 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.535569 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2694a3ee-0897-4753-bfee-ff58550df666" containerName="config-reloader" Jul 21 02:12:15.535595 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.535575 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2694a3ee-0897-4753-bfee-ff58550df666" containerName="prom-label-proxy" Jul 21 02:12:15.535595 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.535580 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2694a3ee-0897-4753-bfee-ff58550df666" containerName="prom-label-proxy" Jul 21 02:12:15.535595 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.535586 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f950068-f8f7-424a-9da8-978105d1e367" containerName="thanos-sidecar" Jul 21 02:12:15.535595 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.535591 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f950068-f8f7-424a-9da8-978105d1e367" containerName="thanos-sidecar" Jul 21 02:12:15.535595 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.535598 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2694a3ee-0897-4753-bfee-ff58550df666" containerName="kube-rbac-proxy-metric" Jul 21 02:12:15.536046 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.535606 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2694a3ee-0897-4753-bfee-ff58550df666" containerName="kube-rbac-proxy-metric" Jul 21 02:12:15.536046 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.535615 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f950068-f8f7-424a-9da8-978105d1e367" containerName="init-config-reloader" Jul 21 02:12:15.536046 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.535622 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f950068-f8f7-424a-9da8-978105d1e367" containerName="init-config-reloader" Jul 21 02:12:15.536046 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.535633 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2694a3ee-0897-4753-bfee-ff58550df666" containerName="kube-rbac-proxy-web" Jul 21 02:12:15.536046 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.535628 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9edf3aea3263a0d1ac1401f349f9d2285dd6c6f5f2f3e092f77bbd4affbe52ba"} err="failed to get container status \"9edf3aea3263a0d1ac1401f349f9d2285dd6c6f5f2f3e092f77bbd4affbe52ba\": rpc error: code = NotFound desc = could not find container \"9edf3aea3263a0d1ac1401f349f9d2285dd6c6f5f2f3e092f77bbd4affbe52ba\": container with ID starting with 9edf3aea3263a0d1ac1401f349f9d2285dd6c6f5f2f3e092f77bbd4affbe52ba not found: ID does not exist" Jul 21 02:12:15.536046 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.535640 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2694a3ee-0897-4753-bfee-ff58550df666" containerName="kube-rbac-proxy-web" Jul 21 02:12:15.536046 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.535659 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2694a3ee-0897-4753-bfee-ff58550df666" containerName="kube-rbac-proxy" Jul 21 02:12:15.536046 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.535667 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2694a3ee-0897-4753-bfee-ff58550df666" containerName="kube-rbac-proxy" Jul 21 02:12:15.536046 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.535674 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f950068-f8f7-424a-9da8-978105d1e367" containerName="kube-rbac-proxy" Jul 21 02:12:15.536046 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.535680 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f950068-f8f7-424a-9da8-978105d1e367" containerName="kube-rbac-proxy" Jul 21 02:12:15.536046 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.535742 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f950068-f8f7-424a-9da8-978105d1e367" containerName="kube-rbac-proxy-thanos" Jul 21 02:12:15.536046 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.535751 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f950068-f8f7-424a-9da8-978105d1e367" containerName="prometheus" Jul 21 02:12:15.536046 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.535758 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2694a3ee-0897-4753-bfee-ff58550df666" containerName="config-reloader" Jul 21 02:12:15.536046 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.535763 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2694a3ee-0897-4753-bfee-ff58550df666" containerName="prom-label-proxy" Jul 21 02:12:15.536046 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.535770 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2694a3ee-0897-4753-bfee-ff58550df666" containerName="kube-rbac-proxy-metric" Jul 21 02:12:15.536046 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.535776 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f950068-f8f7-424a-9da8-978105d1e367" containerName="kube-rbac-proxy-web" Jul 21 02:12:15.536046 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.535783 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f950068-f8f7-424a-9da8-978105d1e367" containerName="thanos-sidecar" Jul 21 02:12:15.536046 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.535788 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2694a3ee-0897-4753-bfee-ff58550df666" containerName="alertmanager" Jul 21 02:12:15.536046 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.535793 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2694a3ee-0897-4753-bfee-ff58550df666" containerName="kube-rbac-proxy-web" Jul 21 02:12:15.536046 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.535798 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f950068-f8f7-424a-9da8-978105d1e367" containerName="config-reloader" Jul 21 02:12:15.536046 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.535803 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2694a3ee-0897-4753-bfee-ff58550df666" containerName="kube-rbac-proxy" Jul 21 02:12:15.536046 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.535808 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f950068-f8f7-424a-9da8-978105d1e367" containerName="kube-rbac-proxy" Jul 21 02:12:15.537871 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.537857 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:12:15.541014 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.540994 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 21 02:12:15.541109 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.540995 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 21 02:12:15.541109 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.541001 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 21 02:12:15.541109 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.541054 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 21 02:12:15.541466 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.541447 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 21 02:12:15.541545 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.541505 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 21 02:12:15.541748 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.541728 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 21 02:12:15.541822 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.541747 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 21 02:12:15.541822 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.541769 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 21 02:12:15.541915 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.541857 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 21 02:12:15.541915 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.541859 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-f6pls\"" Jul 21 02:12:15.542592 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.542570 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 21 02:12:15.542707 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.542683 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-2m0m0vrpk3ssi\"" Jul 21 02:12:15.542707 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.542684 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 21 02:12:15.545410 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.545387 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 21 02:12:15.549786 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.549768 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 21 02:12:15.557257 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.557240 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 21 02:12:15.608863 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.608837 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b26a8446-b697-49ec-a44f-bf5ac51e60a3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b26a8446-b697-49ec-a44f-bf5ac51e60a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:12:15.608962 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.608866 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b26a8446-b697-49ec-a44f-bf5ac51e60a3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b26a8446-b697-49ec-a44f-bf5ac51e60a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:12:15.608962 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.608887 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b26a8446-b697-49ec-a44f-bf5ac51e60a3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b26a8446-b697-49ec-a44f-bf5ac51e60a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:12:15.608962 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.608905 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b26a8446-b697-49ec-a44f-bf5ac51e60a3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b26a8446-b697-49ec-a44f-bf5ac51e60a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:12:15.608962 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.608953 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b26a8446-b697-49ec-a44f-bf5ac51e60a3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b26a8446-b697-49ec-a44f-bf5ac51e60a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:12:15.609121 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.608971 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b26a8446-b697-49ec-a44f-bf5ac51e60a3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b26a8446-b697-49ec-a44f-bf5ac51e60a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:12:15.609121 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.608984 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b26a8446-b697-49ec-a44f-bf5ac51e60a3-config\") pod \"prometheus-k8s-0\" (UID: \"b26a8446-b697-49ec-a44f-bf5ac51e60a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:12:15.609121 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.609033 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b26a8446-b697-49ec-a44f-bf5ac51e60a3-config-out\") pod \"prometheus-k8s-0\" (UID: \"b26a8446-b697-49ec-a44f-bf5ac51e60a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:12:15.609121 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.609048 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skz7m\" (UniqueName: \"kubernetes.io/projected/b26a8446-b697-49ec-a44f-bf5ac51e60a3-kube-api-access-skz7m\") pod \"prometheus-k8s-0\" (UID: \"b26a8446-b697-49ec-a44f-bf5ac51e60a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:12:15.609121 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.609063 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b26a8446-b697-49ec-a44f-bf5ac51e60a3-web-config\") pod \"prometheus-k8s-0\" (UID: \"b26a8446-b697-49ec-a44f-bf5ac51e60a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:12:15.609303 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.609122 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b26a8446-b697-49ec-a44f-bf5ac51e60a3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b26a8446-b697-49ec-a44f-bf5ac51e60a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:12:15.609303 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.609140 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b26a8446-b697-49ec-a44f-bf5ac51e60a3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b26a8446-b697-49ec-a44f-bf5ac51e60a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:12:15.609303 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.609158 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b26a8446-b697-49ec-a44f-bf5ac51e60a3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b26a8446-b697-49ec-a44f-bf5ac51e60a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:12:15.609303 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.609178 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b26a8446-b697-49ec-a44f-bf5ac51e60a3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b26a8446-b697-49ec-a44f-bf5ac51e60a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:12:15.609303 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.609210 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b26a8446-b697-49ec-a44f-bf5ac51e60a3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b26a8446-b697-49ec-a44f-bf5ac51e60a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:12:15.609303 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.609223 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b26a8446-b697-49ec-a44f-bf5ac51e60a3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b26a8446-b697-49ec-a44f-bf5ac51e60a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:12:15.609303 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.609244 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b26a8446-b697-49ec-a44f-bf5ac51e60a3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b26a8446-b697-49ec-a44f-bf5ac51e60a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:12:15.609303 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.609284 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b26a8446-b697-49ec-a44f-bf5ac51e60a3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b26a8446-b697-49ec-a44f-bf5ac51e60a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:12:15.609533 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.609316 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b26a8446-b697-49ec-a44f-bf5ac51e60a3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b26a8446-b697-49ec-a44f-bf5ac51e60a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:12:15.609533 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.609329 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b26a8446-b697-49ec-a44f-bf5ac51e60a3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b26a8446-b697-49ec-a44f-bf5ac51e60a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:12:15.710564 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.710529 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b26a8446-b697-49ec-a44f-bf5ac51e60a3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b26a8446-b697-49ec-a44f-bf5ac51e60a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:12:15.710564 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.710566 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b26a8446-b697-49ec-a44f-bf5ac51e60a3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b26a8446-b697-49ec-a44f-bf5ac51e60a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:12:15.710705 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.710586 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b26a8446-b697-49ec-a44f-bf5ac51e60a3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b26a8446-b697-49ec-a44f-bf5ac51e60a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:12:15.710705 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.710603 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b26a8446-b697-49ec-a44f-bf5ac51e60a3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b26a8446-b697-49ec-a44f-bf5ac51e60a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:12:15.710705 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.710627 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b26a8446-b697-49ec-a44f-bf5ac51e60a3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b26a8446-b697-49ec-a44f-bf5ac51e60a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:12:15.710705 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.710651 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b26a8446-b697-49ec-a44f-bf5ac51e60a3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b26a8446-b697-49ec-a44f-bf5ac51e60a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:12:15.710705 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.710691 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b26a8446-b697-49ec-a44f-bf5ac51e60a3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b26a8446-b697-49ec-a44f-bf5ac51e60a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:12:15.710899 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.710752 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b26a8446-b697-49ec-a44f-bf5ac51e60a3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b26a8446-b697-49ec-a44f-bf5ac51e60a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:12:15.710899 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.710809 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b26a8446-b697-49ec-a44f-bf5ac51e60a3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b26a8446-b697-49ec-a44f-bf5ac51e60a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:12:15.710899 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.710841 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b26a8446-b697-49ec-a44f-bf5ac51e60a3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b26a8446-b697-49ec-a44f-bf5ac51e60a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:12:15.710899 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.710873 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b26a8446-b697-49ec-a44f-bf5ac51e60a3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b26a8446-b697-49ec-a44f-bf5ac51e60a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:12:15.711074 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.710901 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b26a8446-b697-49ec-a44f-bf5ac51e60a3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b26a8446-b697-49ec-a44f-bf5ac51e60a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:12:15.711074 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.710930 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b26a8446-b697-49ec-a44f-bf5ac51e60a3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b26a8446-b697-49ec-a44f-bf5ac51e60a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:12:15.711074 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.710960 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b26a8446-b697-49ec-a44f-bf5ac51e60a3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b26a8446-b697-49ec-a44f-bf5ac51e60a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:12:15.711074 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.710985 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b26a8446-b697-49ec-a44f-bf5ac51e60a3-config\") pod \"prometheus-k8s-0\" (UID: \"b26a8446-b697-49ec-a44f-bf5ac51e60a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:12:15.711074 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.711021 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b26a8446-b697-49ec-a44f-bf5ac51e60a3-config-out\") pod \"prometheus-k8s-0\" (UID: \"b26a8446-b697-49ec-a44f-bf5ac51e60a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:12:15.711074 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.711043 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skz7m\" (UniqueName: \"kubernetes.io/projected/b26a8446-b697-49ec-a44f-bf5ac51e60a3-kube-api-access-skz7m\") pod \"prometheus-k8s-0\" (UID: \"b26a8446-b697-49ec-a44f-bf5ac51e60a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:12:15.711074 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.711070 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b26a8446-b697-49ec-a44f-bf5ac51e60a3-web-config\") pod \"prometheus-k8s-0\" (UID: \"b26a8446-b697-49ec-a44f-bf5ac51e60a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:12:15.711436 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.711109 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b26a8446-b697-49ec-a44f-bf5ac51e60a3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b26a8446-b697-49ec-a44f-bf5ac51e60a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:12:15.711436 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.711135 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b26a8446-b697-49ec-a44f-bf5ac51e60a3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b26a8446-b697-49ec-a44f-bf5ac51e60a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:12:15.711436 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.711374 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b26a8446-b697-49ec-a44f-bf5ac51e60a3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b26a8446-b697-49ec-a44f-bf5ac51e60a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:12:15.711586 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.711454 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b26a8446-b697-49ec-a44f-bf5ac51e60a3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b26a8446-b697-49ec-a44f-bf5ac51e60a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:12:15.711586 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.711557 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b26a8446-b697-49ec-a44f-bf5ac51e60a3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b26a8446-b697-49ec-a44f-bf5ac51e60a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:12:15.711691 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.710958 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b26a8446-b697-49ec-a44f-bf5ac51e60a3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b26a8446-b697-49ec-a44f-bf5ac51e60a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:12:15.711743 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.711711 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b26a8446-b697-49ec-a44f-bf5ac51e60a3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b26a8446-b697-49ec-a44f-bf5ac51e60a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:12:15.712172 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.712151 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b26a8446-b697-49ec-a44f-bf5ac51e60a3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b26a8446-b697-49ec-a44f-bf5ac51e60a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:12:15.713117 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.713096 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b26a8446-b697-49ec-a44f-bf5ac51e60a3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b26a8446-b697-49ec-a44f-bf5ac51e60a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:12:15.713716 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.713689 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b26a8446-b697-49ec-a44f-bf5ac51e60a3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b26a8446-b697-49ec-a44f-bf5ac51e60a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:12:15.713808 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.713736 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b26a8446-b697-49ec-a44f-bf5ac51e60a3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b26a8446-b697-49ec-a44f-bf5ac51e60a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:12:15.713868 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.713816 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b26a8446-b697-49ec-a44f-bf5ac51e60a3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b26a8446-b697-49ec-a44f-bf5ac51e60a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:12:15.714038 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.714014 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b26a8446-b697-49ec-a44f-bf5ac51e60a3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b26a8446-b697-49ec-a44f-bf5ac51e60a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:12:15.714201 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.714178 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b26a8446-b697-49ec-a44f-bf5ac51e60a3-config-out\") pod \"prometheus-k8s-0\" (UID: \"b26a8446-b697-49ec-a44f-bf5ac51e60a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:12:15.714452 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.714432 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b26a8446-b697-49ec-a44f-bf5ac51e60a3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b26a8446-b697-49ec-a44f-bf5ac51e60a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:12:15.714556 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.714525 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b26a8446-b697-49ec-a44f-bf5ac51e60a3-web-config\") pod \"prometheus-k8s-0\" (UID: \"b26a8446-b697-49ec-a44f-bf5ac51e60a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:12:15.715767 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.715753 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b26a8446-b697-49ec-a44f-bf5ac51e60a3-config\") pod \"prometheus-k8s-0\" (UID: \"b26a8446-b697-49ec-a44f-bf5ac51e60a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:12:15.716232 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.716216 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b26a8446-b697-49ec-a44f-bf5ac51e60a3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b26a8446-b697-49ec-a44f-bf5ac51e60a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:12:15.716315 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.716298 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b26a8446-b697-49ec-a44f-bf5ac51e60a3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b26a8446-b697-49ec-a44f-bf5ac51e60a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:12:15.716508 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.716491 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b26a8446-b697-49ec-a44f-bf5ac51e60a3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b26a8446-b697-49ec-a44f-bf5ac51e60a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:12:15.716996 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.716980 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b26a8446-b697-49ec-a44f-bf5ac51e60a3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b26a8446-b697-49ec-a44f-bf5ac51e60a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:12:15.720446 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.720428 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skz7m\" (UniqueName: \"kubernetes.io/projected/b26a8446-b697-49ec-a44f-bf5ac51e60a3-kube-api-access-skz7m\") pod \"prometheus-k8s-0\" (UID: \"b26a8446-b697-49ec-a44f-bf5ac51e60a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:12:15.847408 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:15.847360 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:12:16.003280 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:16.003231 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 21 02:12:16.006029 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:12:16.005986 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb26a8446_b697_49ec_a44f_bf5ac51e60a3.slice/crio-31ec353f3c10bab7cba50fd77cf6bf4dd97f02fcb71d98e99248300eebf025bd WatchSource:0}: Error finding container 31ec353f3c10bab7cba50fd77cf6bf4dd97f02fcb71d98e99248300eebf025bd: Status 404 returned error can't find the container with id 31ec353f3c10bab7cba50fd77cf6bf4dd97f02fcb71d98e99248300eebf025bd Jul 21 02:12:16.482899 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:16.482866 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f950068-f8f7-424a-9da8-978105d1e367" path="/var/lib/kubelet/pods/0f950068-f8f7-424a-9da8-978105d1e367/volumes" Jul 21 02:12:16.483647 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:16.483627 2574 generic.go:356] "Generic (PLEG): container finished" podID="b26a8446-b697-49ec-a44f-bf5ac51e60a3" containerID="3bb066b30b05e2aa318db0dba880d3701c02f7ab1313df493a82497327074eff" exitCode=0 Jul 21 02:12:16.483733 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:16.483709 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b26a8446-b697-49ec-a44f-bf5ac51e60a3","Type":"ContainerDied","Data":"3bb066b30b05e2aa318db0dba880d3701c02f7ab1313df493a82497327074eff"} Jul 21 02:12:16.483809 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:16.483789 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b26a8446-b697-49ec-a44f-bf5ac51e60a3","Type":"ContainerStarted","Data":"31ec353f3c10bab7cba50fd77cf6bf4dd97f02fcb71d98e99248300eebf025bd"} Jul 21 02:12:17.490523 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:17.490484 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b26a8446-b697-49ec-a44f-bf5ac51e60a3","Type":"ContainerStarted","Data":"237ec091f464c8feadbb006d32aa81cf390f79d058dfbc82da733d71c91a020c"} Jul 21 02:12:17.490881 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:17.490574 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b26a8446-b697-49ec-a44f-bf5ac51e60a3","Type":"ContainerStarted","Data":"48142d3e4f3d0e0966bb844f78b9f1f51a6ed47d530b7d0dadd12c52d436da62"} Jul 21 02:12:17.490881 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:17.490603 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b26a8446-b697-49ec-a44f-bf5ac51e60a3","Type":"ContainerStarted","Data":"47c0ca4a6ca3feef63c590bcf6c134faafcd28e11bf0f28dacd2602fae52af8a"} Jul 21 02:12:17.490881 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:17.490627 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b26a8446-b697-49ec-a44f-bf5ac51e60a3","Type":"ContainerStarted","Data":"7a4d0728cdfca0b52a4129ba4ed978b6008a4c61d8f003b182da83cf2e26202b"} Jul 21 02:12:17.490881 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:17.490658 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b26a8446-b697-49ec-a44f-bf5ac51e60a3","Type":"ContainerStarted","Data":"9c72fc2c92135336849da4a58c05f0a11974b09f6cdcd3b8eda65cfb308bf12b"} Jul 21 02:12:17.490881 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:17.490685 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b26a8446-b697-49ec-a44f-bf5ac51e60a3","Type":"ContainerStarted","Data":"51da79584e1354a53ad8d84dead84220976c311a551d14b013ce38a58f7d6066"} Jul 21 02:12:17.524629 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:17.524580 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.501959769 podStartE2EDuration="2.524565885s" totalImagesPullingTime="0s" totalInitContainerRuntime="22.606116ms" isStatefulPod=false podCreationTimestamp="2026-07-21 02:12:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:12:17.522941906 +0000 UTC m=+171.542435857" watchObservedRunningTime="2026-07-21 02:12:17.524565885 +0000 UTC m=+171.544059788" Jul 21 02:12:20.847970 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:12:20.847887 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:13:15.848220 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:13:15.848186 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:13:15.862940 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:13:15.862916 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:13:16.717338 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:13:16.717313 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 02:13:42.912120 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:13:42.912086 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-ftcgh"] Jul 21 02:13:42.914164 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:13:42.914149 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-ftcgh" Jul 21 02:13:42.916723 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:13:42.916697 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-8t5f7\"" Jul 21 02:13:42.916850 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:13:42.916707 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 21 02:13:42.917646 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:13:42.917633 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 21 02:13:42.923292 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:13:42.923258 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-ftcgh"] Jul 21 02:13:42.999536 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:13:42.999512 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e3658273-74e5-4009-bb46-fc10d387142a-bound-sa-token\") pod \"cert-manager-55b4d954d5-ftcgh\" (UID: \"e3658273-74e5-4009-bb46-fc10d387142a\") " pod="cert-manager/cert-manager-55b4d954d5-ftcgh" Jul 21 02:13:42.999694 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:13:42.999572 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6bhb\" (UniqueName: \"kubernetes.io/projected/e3658273-74e5-4009-bb46-fc10d387142a-kube-api-access-j6bhb\") pod \"cert-manager-55b4d954d5-ftcgh\" (UID: \"e3658273-74e5-4009-bb46-fc10d387142a\") " pod="cert-manager/cert-manager-55b4d954d5-ftcgh" Jul 21 02:13:43.101024 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:13:43.100994 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6bhb\" (UniqueName: \"kubernetes.io/projected/e3658273-74e5-4009-bb46-fc10d387142a-kube-api-access-j6bhb\") pod \"cert-manager-55b4d954d5-ftcgh\" (UID: \"e3658273-74e5-4009-bb46-fc10d387142a\") " pod="cert-manager/cert-manager-55b4d954d5-ftcgh" Jul 21 02:13:43.101162 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:13:43.101057 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e3658273-74e5-4009-bb46-fc10d387142a-bound-sa-token\") pod \"cert-manager-55b4d954d5-ftcgh\" (UID: \"e3658273-74e5-4009-bb46-fc10d387142a\") " pod="cert-manager/cert-manager-55b4d954d5-ftcgh" Jul 21 02:13:43.110658 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:13:43.110637 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e3658273-74e5-4009-bb46-fc10d387142a-bound-sa-token\") pod \"cert-manager-55b4d954d5-ftcgh\" (UID: \"e3658273-74e5-4009-bb46-fc10d387142a\") " pod="cert-manager/cert-manager-55b4d954d5-ftcgh" Jul 21 02:13:43.110920 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:13:43.110899 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6bhb\" (UniqueName: \"kubernetes.io/projected/e3658273-74e5-4009-bb46-fc10d387142a-kube-api-access-j6bhb\") pod \"cert-manager-55b4d954d5-ftcgh\" (UID: \"e3658273-74e5-4009-bb46-fc10d387142a\") " pod="cert-manager/cert-manager-55b4d954d5-ftcgh" Jul 21 02:13:43.236040 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:13:43.236014 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-ftcgh" Jul 21 02:13:43.359641 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:13:43.359611 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-ftcgh"] Jul 21 02:13:43.362759 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:13:43.362723 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode3658273_74e5_4009_bb46_fc10d387142a.slice/crio-9501237d293a1f2dad25501ee91340a079c9cbdd4d772f9047a12032ee0f164b WatchSource:0}: Error finding container 9501237d293a1f2dad25501ee91340a079c9cbdd4d772f9047a12032ee0f164b: Status 404 returned error can't find the container with id 9501237d293a1f2dad25501ee91340a079c9cbdd4d772f9047a12032ee0f164b Jul 21 02:13:43.798620 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:13:43.798589 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-ftcgh" event={"ID":"e3658273-74e5-4009-bb46-fc10d387142a","Type":"ContainerStarted","Data":"9501237d293a1f2dad25501ee91340a079c9cbdd4d772f9047a12032ee0f164b"} Jul 21 02:13:46.817035 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:13:46.816995 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-ftcgh" event={"ID":"e3658273-74e5-4009-bb46-fc10d387142a","Type":"ContainerStarted","Data":"3c76d004506063958564655a5f16dfa6c2d038494a7ed83a859e6a5de7ec5f78"} Jul 21 02:13:46.833811 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:13:46.833761 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-ftcgh" podStartSLOduration=2.067471306 podStartE2EDuration="4.833746336s" totalImagesPullingTime="2.76627503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 02:13:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:13:46.832184348 +0000 UTC m=+260.851678271" watchObservedRunningTime="2026-07-21 02:13:46.833746336 +0000 UTC m=+260.853240240" Jul 21 02:14:06.071552 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:06.071517 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8475997445-2dwbj"] Jul 21 02:14:06.075960 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:06.075940 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8475997445-2dwbj" Jul 21 02:14:06.084699 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:14:06.084673 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"opendatahub-operator-controller-manager-service-cert\" is forbidden: User \"system:node:ip-10-0-141-37.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-37.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" type="*v1.Secret" Jul 21 02:14:06.085644 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:14:06.085621 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"opendatahub-operator-controller-manager-8475997445-2dwbj\" is forbidden: User \"system:node:ip-10-0-141-37.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-37.ec2.internal' and this object" podUID="c1c91659-2ab2-4743-a85f-648bf5b5f5b8" pod="opendatahub/opendatahub-operator-controller-manager-8475997445-2dwbj" Jul 21 02:14:06.085898 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:14:06.085878 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"opendatahub-operator-controller-manager-dockercfg-t8cff\" is forbidden: User \"system:node:ip-10-0-141-37.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-37.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-t8cff\"" type="*v1.Secret" Jul 21 02:14:06.085962 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:14:06.085900 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-141-37.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-37.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" Jul 21 02:14:06.085962 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:14:06.085905 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"opendatahub-operator-controller-webhook-cert\" is forbidden: User \"system:node:ip-10-0-141-37.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-37.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" type="*v1.Secret" Jul 21 02:14:06.085962 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:14:06.085899 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-141-37.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-37.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" Jul 21 02:14:06.131235 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:06.131208 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8475997445-2dwbj"] Jul 21 02:14:06.183315 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:06.183255 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c1c91659-2ab2-4743-a85f-648bf5b5f5b8-webhook-cert\") pod \"opendatahub-operator-controller-manager-8475997445-2dwbj\" (UID: \"c1c91659-2ab2-4743-a85f-648bf5b5f5b8\") " pod="opendatahub/opendatahub-operator-controller-manager-8475997445-2dwbj" Jul 21 02:14:06.183484 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:06.183326 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c1c91659-2ab2-4743-a85f-648bf5b5f5b8-tmp\") pod \"opendatahub-operator-controller-manager-8475997445-2dwbj\" (UID: \"c1c91659-2ab2-4743-a85f-648bf5b5f5b8\") " pod="opendatahub/opendatahub-operator-controller-manager-8475997445-2dwbj" Jul 21 02:14:06.183484 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:06.183361 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c1c91659-2ab2-4743-a85f-648bf5b5f5b8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8475997445-2dwbj\" (UID: \"c1c91659-2ab2-4743-a85f-648bf5b5f5b8\") " pod="opendatahub/opendatahub-operator-controller-manager-8475997445-2dwbj" Jul 21 02:14:06.183484 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:06.183377 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zxxk\" (UniqueName: \"kubernetes.io/projected/c1c91659-2ab2-4743-a85f-648bf5b5f5b8-kube-api-access-8zxxk\") pod \"opendatahub-operator-controller-manager-8475997445-2dwbj\" (UID: \"c1c91659-2ab2-4743-a85f-648bf5b5f5b8\") " pod="opendatahub/opendatahub-operator-controller-manager-8475997445-2dwbj" Jul 21 02:14:06.183581 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:06.183488 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c1c91659-2ab2-4743-a85f-648bf5b5f5b8-manifests\") pod \"opendatahub-operator-controller-manager-8475997445-2dwbj\" (UID: \"c1c91659-2ab2-4743-a85f-648bf5b5f5b8\") " pod="opendatahub/opendatahub-operator-controller-manager-8475997445-2dwbj" Jul 21 02:14:06.284162 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:06.284116 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c1c91659-2ab2-4743-a85f-648bf5b5f5b8-webhook-cert\") pod \"opendatahub-operator-controller-manager-8475997445-2dwbj\" (UID: \"c1c91659-2ab2-4743-a85f-648bf5b5f5b8\") " pod="opendatahub/opendatahub-operator-controller-manager-8475997445-2dwbj" Jul 21 02:14:06.284162 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:06.284166 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c1c91659-2ab2-4743-a85f-648bf5b5f5b8-tmp\") pod \"opendatahub-operator-controller-manager-8475997445-2dwbj\" (UID: \"c1c91659-2ab2-4743-a85f-648bf5b5f5b8\") " pod="opendatahub/opendatahub-operator-controller-manager-8475997445-2dwbj" Jul 21 02:14:06.284418 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:06.284199 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c1c91659-2ab2-4743-a85f-648bf5b5f5b8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8475997445-2dwbj\" (UID: \"c1c91659-2ab2-4743-a85f-648bf5b5f5b8\") " pod="opendatahub/opendatahub-operator-controller-manager-8475997445-2dwbj" Jul 21 02:14:06.284418 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:06.284221 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zxxk\" (UniqueName: \"kubernetes.io/projected/c1c91659-2ab2-4743-a85f-648bf5b5f5b8-kube-api-access-8zxxk\") pod \"opendatahub-operator-controller-manager-8475997445-2dwbj\" (UID: \"c1c91659-2ab2-4743-a85f-648bf5b5f5b8\") " pod="opendatahub/opendatahub-operator-controller-manager-8475997445-2dwbj" Jul 21 02:14:06.284418 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:06.284313 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c1c91659-2ab2-4743-a85f-648bf5b5f5b8-manifests\") pod \"opendatahub-operator-controller-manager-8475997445-2dwbj\" (UID: \"c1c91659-2ab2-4743-a85f-648bf5b5f5b8\") " pod="opendatahub/opendatahub-operator-controller-manager-8475997445-2dwbj" Jul 21 02:14:06.284667 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:06.284650 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c1c91659-2ab2-4743-a85f-648bf5b5f5b8-manifests\") pod \"opendatahub-operator-controller-manager-8475997445-2dwbj\" (UID: \"c1c91659-2ab2-4743-a85f-648bf5b5f5b8\") " pod="opendatahub/opendatahub-operator-controller-manager-8475997445-2dwbj" Jul 21 02:14:06.286402 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:06.286386 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c1c91659-2ab2-4743-a85f-648bf5b5f5b8-tmp\") pod \"opendatahub-operator-controller-manager-8475997445-2dwbj\" (UID: \"c1c91659-2ab2-4743-a85f-648bf5b5f5b8\") " pod="opendatahub/opendatahub-operator-controller-manager-8475997445-2dwbj" Jul 21 02:14:06.994066 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:06.994038 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 21 02:14:07.078626 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:07.078600 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 21 02:14:07.219127 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:07.219098 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-t8cff\"" Jul 21 02:14:07.284581 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:14:07.284493 2574 secret.go:189] Couldn't get secret opendatahub/opendatahub-operator-controller-manager-service-cert: failed to sync secret cache: timed out waiting for the condition Jul 21 02:14:07.284581 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:14:07.284582 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1c91659-2ab2-4743-a85f-648bf5b5f5b8-apiservice-cert podName:c1c91659-2ab2-4743-a85f-648bf5b5f5b8 nodeName:}" failed. No retries permitted until 2026-07-21 02:14:07.784562552 +0000 UTC m=+281.804056435 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "apiservice-cert" (UniqueName: "kubernetes.io/secret/c1c91659-2ab2-4743-a85f-648bf5b5f5b8-apiservice-cert") pod "opendatahub-operator-controller-manager-8475997445-2dwbj" (UID: "c1c91659-2ab2-4743-a85f-648bf5b5f5b8") : failed to sync secret cache: timed out waiting for the condition Jul 21 02:14:07.284758 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:14:07.284506 2574 secret.go:189] Couldn't get secret opendatahub/opendatahub-operator-controller-manager-service-cert: failed to sync secret cache: timed out waiting for the condition Jul 21 02:14:07.284758 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:14:07.284695 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1c91659-2ab2-4743-a85f-648bf5b5f5b8-webhook-cert podName:c1c91659-2ab2-4743-a85f-648bf5b5f5b8 nodeName:}" failed. No retries permitted until 2026-07-21 02:14:07.784672609 +0000 UTC m=+281.804166504 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/c1c91659-2ab2-4743-a85f-648bf5b5f5b8-webhook-cert") pod "opendatahub-operator-controller-manager-8475997445-2dwbj" (UID: "c1c91659-2ab2-4743-a85f-648bf5b5f5b8") : failed to sync secret cache: timed out waiting for the condition Jul 21 02:14:07.388285 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:07.388242 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 21 02:14:07.491791 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:07.491752 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 21 02:14:07.496294 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:07.496248 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zxxk\" (UniqueName: \"kubernetes.io/projected/c1c91659-2ab2-4743-a85f-648bf5b5f5b8-kube-api-access-8zxxk\") pod \"opendatahub-operator-controller-manager-8475997445-2dwbj\" (UID: \"c1c91659-2ab2-4743-a85f-648bf5b5f5b8\") " pod="opendatahub/opendatahub-operator-controller-manager-8475997445-2dwbj" Jul 21 02:14:07.794552 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:07.794520 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c1c91659-2ab2-4743-a85f-648bf5b5f5b8-webhook-cert\") pod \"opendatahub-operator-controller-manager-8475997445-2dwbj\" (UID: \"c1c91659-2ab2-4743-a85f-648bf5b5f5b8\") " pod="opendatahub/opendatahub-operator-controller-manager-8475997445-2dwbj" Jul 21 02:14:07.794704 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:07.794596 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c1c91659-2ab2-4743-a85f-648bf5b5f5b8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8475997445-2dwbj\" (UID: \"c1c91659-2ab2-4743-a85f-648bf5b5f5b8\") " pod="opendatahub/opendatahub-operator-controller-manager-8475997445-2dwbj" Jul 21 02:14:07.796730 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:07.796710 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c1c91659-2ab2-4743-a85f-648bf5b5f5b8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8475997445-2dwbj\" (UID: \"c1c91659-2ab2-4743-a85f-648bf5b5f5b8\") " pod="opendatahub/opendatahub-operator-controller-manager-8475997445-2dwbj" Jul 21 02:14:07.796815 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:07.796757 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c1c91659-2ab2-4743-a85f-648bf5b5f5b8-webhook-cert\") pod \"opendatahub-operator-controller-manager-8475997445-2dwbj\" (UID: \"c1c91659-2ab2-4743-a85f-648bf5b5f5b8\") " pod="opendatahub/opendatahub-operator-controller-manager-8475997445-2dwbj" Jul 21 02:14:07.885219 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:07.885192 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8475997445-2dwbj" Jul 21 02:14:08.019125 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:08.019098 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8475997445-2dwbj"] Jul 21 02:14:08.021135 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:14:08.021105 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1c91659_2ab2_4743_a85f_648bf5b5f5b8.slice/crio-e48f8bc6a459554690e6b5905141ae5afb25cfb1e67306f7f9b9868adc66931e WatchSource:0}: Error finding container e48f8bc6a459554690e6b5905141ae5afb25cfb1e67306f7f9b9868adc66931e: Status 404 returned error can't find the container with id e48f8bc6a459554690e6b5905141ae5afb25cfb1e67306f7f9b9868adc66931e Jul 21 02:14:08.900703 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:08.900665 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8475997445-2dwbj" event={"ID":"c1c91659-2ab2-4743-a85f-648bf5b5f5b8","Type":"ContainerStarted","Data":"e48f8bc6a459554690e6b5905141ae5afb25cfb1e67306f7f9b9868adc66931e"} Jul 21 02:14:11.915349 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:11.915317 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8475997445-2dwbj" event={"ID":"c1c91659-2ab2-4743-a85f-648bf5b5f5b8","Type":"ContainerStarted","Data":"979285ed6fc8f5e5979f0ce433de97289064f80977ee759b8be4836c40cad4e4"} Jul 21 02:14:12.920059 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:12.920028 2574 generic.go:356] "Generic (PLEG): container finished" podID="c1c91659-2ab2-4743-a85f-648bf5b5f5b8" containerID="979285ed6fc8f5e5979f0ce433de97289064f80977ee759b8be4836c40cad4e4" exitCode=0 Jul 21 02:14:12.920479 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:12.920098 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8475997445-2dwbj" event={"ID":"c1c91659-2ab2-4743-a85f-648bf5b5f5b8","Type":"ContainerDied","Data":"979285ed6fc8f5e5979f0ce433de97289064f80977ee759b8be4836c40cad4e4"} Jul 21 02:14:13.926259 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:13.926225 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8475997445-2dwbj" event={"ID":"c1c91659-2ab2-4743-a85f-648bf5b5f5b8","Type":"ContainerStarted","Data":"274df7da617b12df9537e373055a02fbdcece3b11f218fdcc75d70433d2ac4a2"} Jul 21 02:14:13.926648 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:13.926356 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-8475997445-2dwbj" Jul 21 02:14:13.962197 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:13.962145 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-8475997445-2dwbj" podStartSLOduration=3.795016637 podStartE2EDuration="7.962128281s" totalImagesPullingTime="3.167142003s" totalInitContainerRuntime="999.969641ms" isStatefulPod=false podCreationTimestamp="2026-07-21 02:14:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:14:13.958901326 +0000 UTC m=+287.978395249" watchObservedRunningTime="2026-07-21 02:14:13.962128281 +0000 UTC m=+287.981622186" Jul 21 02:14:24.932853 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:24.932823 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-8475997445-2dwbj" Jul 21 02:14:27.362614 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:27.362579 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-2pth6"] Jul 21 02:14:27.366123 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:27.366101 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-2pth6" Jul 21 02:14:27.371361 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:27.371344 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-s89vw\"" Jul 21 02:14:27.391786 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:27.391767 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 21 02:14:27.410310 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:27.410241 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-2pth6"] Jul 21 02:14:27.443067 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:27.443038 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/21493e6e-e2a6-41cc-8483-a644130919a2-cert\") pod \"odh-model-controller-7998c67dc4-2pth6\" (UID: \"21493e6e-e2a6-41cc-8483-a644130919a2\") " pod="opendatahub/odh-model-controller-7998c67dc4-2pth6" Jul 21 02:14:27.443210 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:27.443072 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk582\" (UniqueName: \"kubernetes.io/projected/21493e6e-e2a6-41cc-8483-a644130919a2-kube-api-access-dk582\") pod \"odh-model-controller-7998c67dc4-2pth6\" (UID: \"21493e6e-e2a6-41cc-8483-a644130919a2\") " pod="opendatahub/odh-model-controller-7998c67dc4-2pth6" Jul 21 02:14:27.544139 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:27.544107 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/21493e6e-e2a6-41cc-8483-a644130919a2-cert\") pod \"odh-model-controller-7998c67dc4-2pth6\" (UID: \"21493e6e-e2a6-41cc-8483-a644130919a2\") " pod="opendatahub/odh-model-controller-7998c67dc4-2pth6" Jul 21 02:14:27.544139 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:27.544138 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk582\" (UniqueName: \"kubernetes.io/projected/21493e6e-e2a6-41cc-8483-a644130919a2-kube-api-access-dk582\") pod \"odh-model-controller-7998c67dc4-2pth6\" (UID: \"21493e6e-e2a6-41cc-8483-a644130919a2\") " pod="opendatahub/odh-model-controller-7998c67dc4-2pth6" Jul 21 02:14:27.544361 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:14:27.544288 2574 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 21 02:14:27.544405 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:14:27.544382 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21493e6e-e2a6-41cc-8483-a644130919a2-cert podName:21493e6e-e2a6-41cc-8483-a644130919a2 nodeName:}" failed. No retries permitted until 2026-07-21 02:14:28.044365049 +0000 UTC m=+302.063858932 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/21493e6e-e2a6-41cc-8483-a644130919a2-cert") pod "odh-model-controller-7998c67dc4-2pth6" (UID: "21493e6e-e2a6-41cc-8483-a644130919a2") : secret "odh-model-controller-webhook-cert" not found Jul 21 02:14:27.556626 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:27.556599 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk582\" (UniqueName: \"kubernetes.io/projected/21493e6e-e2a6-41cc-8483-a644130919a2-kube-api-access-dk582\") pod \"odh-model-controller-7998c67dc4-2pth6\" (UID: \"21493e6e-e2a6-41cc-8483-a644130919a2\") " pod="opendatahub/odh-model-controller-7998c67dc4-2pth6" Jul 21 02:14:28.048194 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:28.048159 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/21493e6e-e2a6-41cc-8483-a644130919a2-cert\") pod \"odh-model-controller-7998c67dc4-2pth6\" (UID: \"21493e6e-e2a6-41cc-8483-a644130919a2\") " pod="opendatahub/odh-model-controller-7998c67dc4-2pth6" Jul 21 02:14:28.050402 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:28.050381 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/21493e6e-e2a6-41cc-8483-a644130919a2-cert\") pod \"odh-model-controller-7998c67dc4-2pth6\" (UID: \"21493e6e-e2a6-41cc-8483-a644130919a2\") " pod="opendatahub/odh-model-controller-7998c67dc4-2pth6" Jul 21 02:14:28.139644 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:28.139609 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-mjtvp"] Jul 21 02:14:28.143416 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:28.143396 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-mjtvp" Jul 21 02:14:28.146976 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:28.146951 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 21 02:14:28.157242 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:28.157218 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-mjtvp"] Jul 21 02:14:28.249847 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:28.249814 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bvqw\" (UniqueName: \"kubernetes.io/projected/7126c66e-8b6b-4a9b-b93f-62f9d393ad72-kube-api-access-4bvqw\") pod \"model-serving-api-54d89bd5fc-mjtvp\" (UID: \"7126c66e-8b6b-4a9b-b93f-62f9d393ad72\") " pod="opendatahub/model-serving-api-54d89bd5fc-mjtvp" Jul 21 02:14:28.250013 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:28.249854 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7126c66e-8b6b-4a9b-b93f-62f9d393ad72-tls-certs\") pod \"model-serving-api-54d89bd5fc-mjtvp\" (UID: \"7126c66e-8b6b-4a9b-b93f-62f9d393ad72\") " pod="opendatahub/model-serving-api-54d89bd5fc-mjtvp" Jul 21 02:14:28.279175 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:28.279150 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-s89vw\"" Jul 21 02:14:28.287054 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:28.287016 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-2pth6" Jul 21 02:14:28.313289 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:28.313237 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qv9kr_bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9/ovn-acl-logging/0.log" Jul 21 02:14:28.315290 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:28.315242 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qv9kr_bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9/ovn-acl-logging/0.log" Jul 21 02:14:28.322060 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:28.322033 2574 kubelet.go:1648] "Image garbage collection succeeded" Jul 21 02:14:28.350789 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:28.350757 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bvqw\" (UniqueName: \"kubernetes.io/projected/7126c66e-8b6b-4a9b-b93f-62f9d393ad72-kube-api-access-4bvqw\") pod \"model-serving-api-54d89bd5fc-mjtvp\" (UID: \"7126c66e-8b6b-4a9b-b93f-62f9d393ad72\") " pod="opendatahub/model-serving-api-54d89bd5fc-mjtvp" Jul 21 02:14:28.350912 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:28.350821 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7126c66e-8b6b-4a9b-b93f-62f9d393ad72-tls-certs\") pod \"model-serving-api-54d89bd5fc-mjtvp\" (UID: \"7126c66e-8b6b-4a9b-b93f-62f9d393ad72\") " pod="opendatahub/model-serving-api-54d89bd5fc-mjtvp" Jul 21 02:14:28.353772 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:28.353753 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 21 02:14:28.360434 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:28.360403 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bvqw\" (UniqueName: \"kubernetes.io/projected/7126c66e-8b6b-4a9b-b93f-62f9d393ad72-kube-api-access-4bvqw\") pod \"model-serving-api-54d89bd5fc-mjtvp\" (UID: \"7126c66e-8b6b-4a9b-b93f-62f9d393ad72\") " pod="opendatahub/model-serving-api-54d89bd5fc-mjtvp" Jul 21 02:14:28.362124 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:14:28.361891 2574 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 21 02:14:28.362124 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:14:28.361969 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7126c66e-8b6b-4a9b-b93f-62f9d393ad72-tls-certs podName:7126c66e-8b6b-4a9b-b93f-62f9d393ad72 nodeName:}" failed. No retries permitted until 2026-07-21 02:14:28.861953702 +0000 UTC m=+302.881447584 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/7126c66e-8b6b-4a9b-b93f-62f9d393ad72-tls-certs") pod "model-serving-api-54d89bd5fc-mjtvp" (UID: "7126c66e-8b6b-4a9b-b93f-62f9d393ad72") : secret "model-serving-api-tls" not found Jul 21 02:14:28.436986 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:28.436751 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-2pth6"] Jul 21 02:14:28.955893 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:28.955855 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7126c66e-8b6b-4a9b-b93f-62f9d393ad72-tls-certs\") pod \"model-serving-api-54d89bd5fc-mjtvp\" (UID: \"7126c66e-8b6b-4a9b-b93f-62f9d393ad72\") " pod="opendatahub/model-serving-api-54d89bd5fc-mjtvp" Jul 21 02:14:28.958184 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:28.958161 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7126c66e-8b6b-4a9b-b93f-62f9d393ad72-tls-certs\") pod \"model-serving-api-54d89bd5fc-mjtvp\" (UID: \"7126c66e-8b6b-4a9b-b93f-62f9d393ad72\") " pod="opendatahub/model-serving-api-54d89bd5fc-mjtvp" Jul 21 02:14:28.989878 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:28.989838 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-2pth6" event={"ID":"21493e6e-e2a6-41cc-8483-a644130919a2","Type":"ContainerStarted","Data":"261172d57626f41beb66ae9510438b1c36879d5e1428a1a3cd709976fd75a00b"} Jul 21 02:14:29.053838 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:29.053809 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-mjtvp" Jul 21 02:14:29.181930 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:29.181749 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-mjtvp"] Jul 21 02:14:29.184566 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:14:29.184527 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7126c66e_8b6b_4a9b_b93f_62f9d393ad72.slice/crio-4bbd5f67cd09386cbeed212d7bff6f406930fb3895d5a2995fcf65bacad40afe WatchSource:0}: Error finding container 4bbd5f67cd09386cbeed212d7bff6f406930fb3895d5a2995fcf65bacad40afe: Status 404 returned error can't find the container with id 4bbd5f67cd09386cbeed212d7bff6f406930fb3895d5a2995fcf65bacad40afe Jul 21 02:14:29.996828 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:29.996792 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-mjtvp" event={"ID":"7126c66e-8b6b-4a9b-b93f-62f9d393ad72","Type":"ContainerStarted","Data":"4bbd5f67cd09386cbeed212d7bff6f406930fb3895d5a2995fcf65bacad40afe"} Jul 21 02:14:32.007744 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:32.007703 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-2pth6" event={"ID":"21493e6e-e2a6-41cc-8483-a644130919a2","Type":"ContainerStarted","Data":"cee43b6c3a038b858f535bd2b0ee3a66c22e905d0d470f4f8a04d24d245aeadc"} Jul 21 02:14:32.008162 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:32.007802 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-2pth6" Jul 21 02:14:32.009199 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:32.009175 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-mjtvp" event={"ID":"7126c66e-8b6b-4a9b-b93f-62f9d393ad72","Type":"ContainerStarted","Data":"951d0452c70cff233a1de5d7e693ee5b4179c8113d16b091ebfc9ea5e5dfb103"} Jul 21 02:14:32.009337 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:32.009320 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-mjtvp" Jul 21 02:14:32.026296 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:32.026219 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-2pth6" podStartSLOduration=1.8322728609999999 podStartE2EDuration="5.026206422s" totalImagesPullingTime="3.193933561s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 02:14:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:14:32.025223099 +0000 UTC m=+306.044716998" watchObservedRunningTime="2026-07-21 02:14:32.026206422 +0000 UTC m=+306.045700325" Jul 21 02:14:32.043357 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:32.043305 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-mjtvp" podStartSLOduration=1.595257122 podStartE2EDuration="4.043291892s" totalImagesPullingTime="2.44803477s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 02:14:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:14:32.043043773 +0000 UTC m=+306.062537700" watchObservedRunningTime="2026-07-21 02:14:32.043291892 +0000 UTC m=+306.062785787" Jul 21 02:14:37.553206 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:37.553162 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-9x9k7"] Jul 21 02:14:37.555966 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:37.555945 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9x9k7" Jul 21 02:14:37.561916 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:37.561896 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 21 02:14:37.562079 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:37.562060 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-jv98j\"" Jul 21 02:14:37.576802 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:37.576783 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-9x9k7"] Jul 21 02:14:37.598317 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:37.598294 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-pwzb2"] Jul 21 02:14:37.606807 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:37.606786 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pwzb2" Jul 21 02:14:37.613616 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:37.613597 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 21 02:14:37.613616 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:37.613609 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-5lq5f\"" Jul 21 02:14:37.630056 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:37.630031 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b0e21853-e493-4e53-9b6f-fdc88f1d766f-cert\") pod \"kserve-controller-manager-d6f4b65b8-9x9k7\" (UID: \"b0e21853-e493-4e53-9b6f-fdc88f1d766f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-9x9k7" Jul 21 02:14:37.630188 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:37.630172 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz4rv\" (UniqueName: \"kubernetes.io/projected/b0e21853-e493-4e53-9b6f-fdc88f1d766f-kube-api-access-fz4rv\") pod \"kserve-controller-manager-d6f4b65b8-9x9k7\" (UID: \"b0e21853-e493-4e53-9b6f-fdc88f1d766f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-9x9k7" Jul 21 02:14:37.641933 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:37.641913 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-pwzb2"] Jul 21 02:14:37.730856 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:37.730820 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pntk\" (UniqueName: \"kubernetes.io/projected/54ddeee7-7991-44cb-9c3a-dc445f966bb2-kube-api-access-2pntk\") pod \"llmisvc-controller-manager-777c6bd7d5-pwzb2\" (UID: \"54ddeee7-7991-44cb-9c3a-dc445f966bb2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pwzb2" Jul 21 02:14:37.731039 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:37.730867 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54ddeee7-7991-44cb-9c3a-dc445f966bb2-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-pwzb2\" (UID: \"54ddeee7-7991-44cb-9c3a-dc445f966bb2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pwzb2" Jul 21 02:14:37.731039 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:37.730927 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b0e21853-e493-4e53-9b6f-fdc88f1d766f-cert\") pod \"kserve-controller-manager-d6f4b65b8-9x9k7\" (UID: \"b0e21853-e493-4e53-9b6f-fdc88f1d766f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-9x9k7" Jul 21 02:14:37.731039 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:37.730966 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz4rv\" (UniqueName: \"kubernetes.io/projected/b0e21853-e493-4e53-9b6f-fdc88f1d766f-kube-api-access-fz4rv\") pod \"kserve-controller-manager-d6f4b65b8-9x9k7\" (UID: \"b0e21853-e493-4e53-9b6f-fdc88f1d766f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-9x9k7" Jul 21 02:14:37.733799 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:37.733776 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b0e21853-e493-4e53-9b6f-fdc88f1d766f-cert\") pod \"kserve-controller-manager-d6f4b65b8-9x9k7\" (UID: \"b0e21853-e493-4e53-9b6f-fdc88f1d766f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-9x9k7" Jul 21 02:14:37.755286 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:37.753806 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz4rv\" (UniqueName: \"kubernetes.io/projected/b0e21853-e493-4e53-9b6f-fdc88f1d766f-kube-api-access-fz4rv\") pod \"kserve-controller-manager-d6f4b65b8-9x9k7\" (UID: \"b0e21853-e493-4e53-9b6f-fdc88f1d766f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-9x9k7" Jul 21 02:14:37.832418 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:37.832327 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pntk\" (UniqueName: \"kubernetes.io/projected/54ddeee7-7991-44cb-9c3a-dc445f966bb2-kube-api-access-2pntk\") pod \"llmisvc-controller-manager-777c6bd7d5-pwzb2\" (UID: \"54ddeee7-7991-44cb-9c3a-dc445f966bb2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pwzb2" Jul 21 02:14:37.832418 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:37.832368 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54ddeee7-7991-44cb-9c3a-dc445f966bb2-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-pwzb2\" (UID: \"54ddeee7-7991-44cb-9c3a-dc445f966bb2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pwzb2" Jul 21 02:14:37.834643 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:37.834620 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54ddeee7-7991-44cb-9c3a-dc445f966bb2-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-pwzb2\" (UID: \"54ddeee7-7991-44cb-9c3a-dc445f966bb2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pwzb2" Jul 21 02:14:37.866341 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:37.866312 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9x9k7" Jul 21 02:14:37.870799 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:37.870769 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pntk\" (UniqueName: \"kubernetes.io/projected/54ddeee7-7991-44cb-9c3a-dc445f966bb2-kube-api-access-2pntk\") pod \"llmisvc-controller-manager-777c6bd7d5-pwzb2\" (UID: \"54ddeee7-7991-44cb-9c3a-dc445f966bb2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pwzb2" Jul 21 02:14:37.895171 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:37.895137 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5d4c7dc87d-52tfh"] Jul 21 02:14:37.899605 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:37.899580 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5d4c7dc87d-52tfh" Jul 21 02:14:37.909014 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:37.908463 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 21 02:14:37.909014 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:37.908768 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 21 02:14:37.909014 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:37.908983 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 21 02:14:37.909314 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:37.909082 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 21 02:14:37.915767 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:37.915742 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pwzb2" Jul 21 02:14:37.922396 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:37.922318 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5d4c7dc87d-52tfh"] Jul 21 02:14:38.036978 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:38.036948 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5qfg\" (UniqueName: \"kubernetes.io/projected/e2d7f3ea-52ce-4b44-9821-6f34b323f0f9-kube-api-access-b5qfg\") pod \"kube-auth-proxy-5d4c7dc87d-52tfh\" (UID: \"e2d7f3ea-52ce-4b44-9821-6f34b323f0f9\") " pod="openshift-ingress/kube-auth-proxy-5d4c7dc87d-52tfh" Jul 21 02:14:38.036978 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:38.036982 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e2d7f3ea-52ce-4b44-9821-6f34b323f0f9-tmp\") pod \"kube-auth-proxy-5d4c7dc87d-52tfh\" (UID: \"e2d7f3ea-52ce-4b44-9821-6f34b323f0f9\") " pod="openshift-ingress/kube-auth-proxy-5d4c7dc87d-52tfh" Jul 21 02:14:38.037209 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:38.037002 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2d7f3ea-52ce-4b44-9821-6f34b323f0f9-tls-certs\") pod \"kube-auth-proxy-5d4c7dc87d-52tfh\" (UID: \"e2d7f3ea-52ce-4b44-9821-6f34b323f0f9\") " pod="openshift-ingress/kube-auth-proxy-5d4c7dc87d-52tfh" Jul 21 02:14:38.042331 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:38.042146 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-9x9k7"] Jul 21 02:14:38.044687 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:14:38.044659 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0e21853_e493_4e53_9b6f_fdc88f1d766f.slice/crio-40d2a4b364cea77943b5633574197755725971c0650aa62f6295ba328cf5ac79 WatchSource:0}: Error finding container 40d2a4b364cea77943b5633574197755725971c0650aa62f6295ba328cf5ac79: Status 404 returned error can't find the container with id 40d2a4b364cea77943b5633574197755725971c0650aa62f6295ba328cf5ac79 Jul 21 02:14:38.046523 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:38.046507 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 02:14:38.102992 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:38.102697 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-pwzb2"] Jul 21 02:14:38.106003 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:14:38.105974 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod54ddeee7_7991_44cb_9c3a_dc445f966bb2.slice/crio-0f94d9c89d659843d130203c2ef87bd6837e95f428635aa39caffce46814f8fe WatchSource:0}: Error finding container 0f94d9c89d659843d130203c2ef87bd6837e95f428635aa39caffce46814f8fe: Status 404 returned error can't find the container with id 0f94d9c89d659843d130203c2ef87bd6837e95f428635aa39caffce46814f8fe Jul 21 02:14:38.137903 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:38.137878 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5qfg\" (UniqueName: \"kubernetes.io/projected/e2d7f3ea-52ce-4b44-9821-6f34b323f0f9-kube-api-access-b5qfg\") pod \"kube-auth-proxy-5d4c7dc87d-52tfh\" (UID: \"e2d7f3ea-52ce-4b44-9821-6f34b323f0f9\") " pod="openshift-ingress/kube-auth-proxy-5d4c7dc87d-52tfh" Jul 21 02:14:38.138035 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:38.137911 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e2d7f3ea-52ce-4b44-9821-6f34b323f0f9-tmp\") pod \"kube-auth-proxy-5d4c7dc87d-52tfh\" (UID: \"e2d7f3ea-52ce-4b44-9821-6f34b323f0f9\") " pod="openshift-ingress/kube-auth-proxy-5d4c7dc87d-52tfh" Jul 21 02:14:38.138035 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:38.137931 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2d7f3ea-52ce-4b44-9821-6f34b323f0f9-tls-certs\") pod \"kube-auth-proxy-5d4c7dc87d-52tfh\" (UID: \"e2d7f3ea-52ce-4b44-9821-6f34b323f0f9\") " pod="openshift-ingress/kube-auth-proxy-5d4c7dc87d-52tfh" Jul 21 02:14:38.140099 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:38.140076 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e2d7f3ea-52ce-4b44-9821-6f34b323f0f9-tmp\") pod \"kube-auth-proxy-5d4c7dc87d-52tfh\" (UID: \"e2d7f3ea-52ce-4b44-9821-6f34b323f0f9\") " pod="openshift-ingress/kube-auth-proxy-5d4c7dc87d-52tfh" Jul 21 02:14:38.140304 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:38.140284 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2d7f3ea-52ce-4b44-9821-6f34b323f0f9-tls-certs\") pod \"kube-auth-proxy-5d4c7dc87d-52tfh\" (UID: \"e2d7f3ea-52ce-4b44-9821-6f34b323f0f9\") " pod="openshift-ingress/kube-auth-proxy-5d4c7dc87d-52tfh" Jul 21 02:14:38.152986 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:38.152963 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5qfg\" (UniqueName: \"kubernetes.io/projected/e2d7f3ea-52ce-4b44-9821-6f34b323f0f9-kube-api-access-b5qfg\") pod \"kube-auth-proxy-5d4c7dc87d-52tfh\" (UID: \"e2d7f3ea-52ce-4b44-9821-6f34b323f0f9\") " pod="openshift-ingress/kube-auth-proxy-5d4c7dc87d-52tfh" Jul 21 02:14:38.210052 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:38.210030 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5d4c7dc87d-52tfh" Jul 21 02:14:38.358078 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:38.357991 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5d4c7dc87d-52tfh"] Jul 21 02:14:38.360527 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:14:38.360501 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2d7f3ea_52ce_4b44_9821_6f34b323f0f9.slice/crio-51184af30e0e24f389901ccf11f8e300da554dd033d35405386d58783e229e2f WatchSource:0}: Error finding container 51184af30e0e24f389901ccf11f8e300da554dd033d35405386d58783e229e2f: Status 404 returned error can't find the container with id 51184af30e0e24f389901ccf11f8e300da554dd033d35405386d58783e229e2f Jul 21 02:14:39.040439 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:39.040405 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pwzb2" event={"ID":"54ddeee7-7991-44cb-9c3a-dc445f966bb2","Type":"ContainerStarted","Data":"0f94d9c89d659843d130203c2ef87bd6837e95f428635aa39caffce46814f8fe"} Jul 21 02:14:39.041465 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:39.041439 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5d4c7dc87d-52tfh" event={"ID":"e2d7f3ea-52ce-4b44-9821-6f34b323f0f9","Type":"ContainerStarted","Data":"51184af30e0e24f389901ccf11f8e300da554dd033d35405386d58783e229e2f"} Jul 21 02:14:39.042413 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:39.042395 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9x9k7" event={"ID":"b0e21853-e493-4e53-9b6f-fdc88f1d766f","Type":"ContainerStarted","Data":"40d2a4b364cea77943b5633574197755725971c0650aa62f6295ba328cf5ac79"} Jul 21 02:14:40.459378 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:40.457790 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2zbzn"] Jul 21 02:14:40.462344 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:40.462319 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2zbzn" Jul 21 02:14:40.466163 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:40.466139 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 21 02:14:40.467183 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:40.467158 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 21 02:14:40.467341 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:40.467312 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 21 02:14:40.467461 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:40.467430 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-rwqq5\"" Jul 21 02:14:40.467636 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:40.467595 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 21 02:14:40.475593 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:40.475572 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2zbzn"] Jul 21 02:14:40.561698 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:40.561645 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cbeff22-f3f1-4e40-85c8-02c7971c9c47-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-2zbzn\" (UID: \"9cbeff22-f3f1-4e40-85c8-02c7971c9c47\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2zbzn" Jul 21 02:14:40.561878 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:40.561782 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9cbeff22-f3f1-4e40-85c8-02c7971c9c47-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-2zbzn\" (UID: \"9cbeff22-f3f1-4e40-85c8-02c7971c9c47\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2zbzn" Jul 21 02:14:40.561951 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:40.561940 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cbeff22-f3f1-4e40-85c8-02c7971c9c47-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-2zbzn\" (UID: \"9cbeff22-f3f1-4e40-85c8-02c7971c9c47\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2zbzn" Jul 21 02:14:40.562226 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:40.562153 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9cbeff22-f3f1-4e40-85c8-02c7971c9c47-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-2zbzn\" (UID: \"9cbeff22-f3f1-4e40-85c8-02c7971c9c47\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2zbzn" Jul 21 02:14:40.562226 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:40.562194 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9cbeff22-f3f1-4e40-85c8-02c7971c9c47-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-2zbzn\" (UID: \"9cbeff22-f3f1-4e40-85c8-02c7971c9c47\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2zbzn" Jul 21 02:14:40.562444 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:40.562244 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9cbeff22-f3f1-4e40-85c8-02c7971c9c47-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-2zbzn\" (UID: \"9cbeff22-f3f1-4e40-85c8-02c7971c9c47\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2zbzn" Jul 21 02:14:40.562444 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:40.562333 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzvk8\" (UniqueName: \"kubernetes.io/projected/9cbeff22-f3f1-4e40-85c8-02c7971c9c47-kube-api-access-tzvk8\") pod \"istiod-openshift-gateway-94bb8fbfd-2zbzn\" (UID: \"9cbeff22-f3f1-4e40-85c8-02c7971c9c47\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2zbzn" Jul 21 02:14:40.663513 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:40.663482 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cbeff22-f3f1-4e40-85c8-02c7971c9c47-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-2zbzn\" (UID: \"9cbeff22-f3f1-4e40-85c8-02c7971c9c47\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2zbzn" Jul 21 02:14:40.663694 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:40.663530 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9cbeff22-f3f1-4e40-85c8-02c7971c9c47-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-2zbzn\" (UID: \"9cbeff22-f3f1-4e40-85c8-02c7971c9c47\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2zbzn" Jul 21 02:14:40.663694 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:40.663564 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9cbeff22-f3f1-4e40-85c8-02c7971c9c47-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-2zbzn\" (UID: \"9cbeff22-f3f1-4e40-85c8-02c7971c9c47\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2zbzn" Jul 21 02:14:40.663694 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:40.663592 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9cbeff22-f3f1-4e40-85c8-02c7971c9c47-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-2zbzn\" (UID: \"9cbeff22-f3f1-4e40-85c8-02c7971c9c47\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2zbzn" Jul 21 02:14:40.663694 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:40.663614 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzvk8\" (UniqueName: \"kubernetes.io/projected/9cbeff22-f3f1-4e40-85c8-02c7971c9c47-kube-api-access-tzvk8\") pod \"istiod-openshift-gateway-94bb8fbfd-2zbzn\" (UID: \"9cbeff22-f3f1-4e40-85c8-02c7971c9c47\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2zbzn" Jul 21 02:14:40.663694 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:40.663671 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cbeff22-f3f1-4e40-85c8-02c7971c9c47-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-2zbzn\" (UID: \"9cbeff22-f3f1-4e40-85c8-02c7971c9c47\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2zbzn" Jul 21 02:14:40.663925 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:40.663718 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9cbeff22-f3f1-4e40-85c8-02c7971c9c47-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-2zbzn\" (UID: \"9cbeff22-f3f1-4e40-85c8-02c7971c9c47\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2zbzn" Jul 21 02:14:40.676798 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:40.671858 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9cbeff22-f3f1-4e40-85c8-02c7971c9c47-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-2zbzn\" (UID: \"9cbeff22-f3f1-4e40-85c8-02c7971c9c47\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2zbzn" Jul 21 02:14:40.679926 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:40.679077 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9cbeff22-f3f1-4e40-85c8-02c7971c9c47-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-2zbzn\" (UID: \"9cbeff22-f3f1-4e40-85c8-02c7971c9c47\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2zbzn" Jul 21 02:14:40.679926 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:40.679412 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9cbeff22-f3f1-4e40-85c8-02c7971c9c47-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-2zbzn\" (UID: \"9cbeff22-f3f1-4e40-85c8-02c7971c9c47\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2zbzn" Jul 21 02:14:40.679926 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:40.679605 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cbeff22-f3f1-4e40-85c8-02c7971c9c47-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-2zbzn\" (UID: \"9cbeff22-f3f1-4e40-85c8-02c7971c9c47\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2zbzn" Jul 21 02:14:40.687626 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:40.687597 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9cbeff22-f3f1-4e40-85c8-02c7971c9c47-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-2zbzn\" (UID: \"9cbeff22-f3f1-4e40-85c8-02c7971c9c47\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2zbzn" Jul 21 02:14:40.721117 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:40.721048 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cbeff22-f3f1-4e40-85c8-02c7971c9c47-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-2zbzn\" (UID: \"9cbeff22-f3f1-4e40-85c8-02c7971c9c47\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2zbzn" Jul 21 02:14:40.721361 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:40.721339 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzvk8\" (UniqueName: \"kubernetes.io/projected/9cbeff22-f3f1-4e40-85c8-02c7971c9c47-kube-api-access-tzvk8\") pod \"istiod-openshift-gateway-94bb8fbfd-2zbzn\" (UID: \"9cbeff22-f3f1-4e40-85c8-02c7971c9c47\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2zbzn" Jul 21 02:14:40.775933 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:40.775905 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2zbzn" Jul 21 02:14:42.290925 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:42.290898 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2zbzn"] Jul 21 02:14:42.292499 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:14:42.292477 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cbeff22_f3f1_4e40_85c8_02c7971c9c47.slice/crio-c2864a4b48f4c9a8008a07f527ca40d9258d21374e71b46a568ad8ea9c552c7b WatchSource:0}: Error finding container c2864a4b48f4c9a8008a07f527ca40d9258d21374e71b46a568ad8ea9c552c7b: Status 404 returned error can't find the container with id c2864a4b48f4c9a8008a07f527ca40d9258d21374e71b46a568ad8ea9c552c7b Jul 21 02:14:43.015203 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:43.015174 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-2pth6" Jul 21 02:14:43.018127 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:43.018102 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-mjtvp" Jul 21 02:14:43.068111 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:43.068075 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pwzb2" event={"ID":"54ddeee7-7991-44cb-9c3a-dc445f966bb2","Type":"ContainerStarted","Data":"b931ce0ac1d7873127be75c700137551facbc00a63e219414f855fc1626da7e6"} Jul 21 02:14:43.068296 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:43.068148 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pwzb2" Jul 21 02:14:43.070468 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:43.070428 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5d4c7dc87d-52tfh" event={"ID":"e2d7f3ea-52ce-4b44-9821-6f34b323f0f9","Type":"ContainerStarted","Data":"4d7aa0e12c71c2d4aef122b7d3fc6169d27f24ac10bb7a32b57277d6c00632fd"} Jul 21 02:14:43.073592 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:43.073545 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9x9k7" event={"ID":"b0e21853-e493-4e53-9b6f-fdc88f1d766f","Type":"ContainerStarted","Data":"21ec7c990f7bcda4ef2e40827374c15c29bf9e7e942ab420bc7123f7aa92b3bb"} Jul 21 02:14:43.077852 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:43.077808 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2zbzn" event={"ID":"9cbeff22-f3f1-4e40-85c8-02c7971c9c47","Type":"ContainerStarted","Data":"c2864a4b48f4c9a8008a07f527ca40d9258d21374e71b46a568ad8ea9c552c7b"} Jul 21 02:14:43.117002 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:43.116381 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9x9k7" podStartSLOduration=2.016219771 podStartE2EDuration="6.116357371s" totalImagesPullingTime="4.1001376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 02:14:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:14:43.116196984 +0000 UTC m=+317.135690889" watchObservedRunningTime="2026-07-21 02:14:43.116357371 +0000 UTC m=+317.135851277" Jul 21 02:14:43.163155 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:43.163092 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pwzb2" podStartSLOduration=2.11728779 podStartE2EDuration="6.163070412s" totalImagesPullingTime="4.045782622s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 02:14:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:14:43.160570685 +0000 UTC m=+317.180064611" watchObservedRunningTime="2026-07-21 02:14:43.163070412 +0000 UTC m=+317.182564320" Jul 21 02:14:43.183648 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:43.183603 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5d4c7dc87d-52tfh" podStartSLOduration=2.398427285 podStartE2EDuration="6.183588258s" totalImagesPullingTime="3.785160973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 02:14:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:14:43.182920605 +0000 UTC m=+317.202414511" watchObservedRunningTime="2026-07-21 02:14:43.183588258 +0000 UTC m=+317.203082160" Jul 21 02:14:44.083500 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:44.083450 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9x9k7" Jul 21 02:14:45.063790 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:45.063754 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 21 02:14:45.063885 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:45.063819 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 21 02:14:46.092588 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:46.092546 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2zbzn" event={"ID":"9cbeff22-f3f1-4e40-85c8-02c7971c9c47","Type":"ContainerStarted","Data":"54c1e112cca6c9d94444c480b3d1fe6d28c71a2b698ef251ec799e3453ace515"} Jul 21 02:14:46.093032 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:46.092756 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2zbzn" Jul 21 02:14:46.094414 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:46.094388 2574 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-2zbzn container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 21 02:14:46.094525 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:46.094438 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2zbzn" podUID="9cbeff22-f3f1-4e40-85c8-02c7971c9c47" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 21 02:14:46.114463 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:46.114418 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2zbzn" podStartSLOduration=3.345343563 podStartE2EDuration="6.114401952s" totalImagesPullingTime="2.769058389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 02:14:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:14:46.113664985 +0000 UTC m=+320.133158888" watchObservedRunningTime="2026-07-21 02:14:46.114401952 +0000 UTC m=+320.133895856" Jul 21 02:14:47.097307 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:14:47.097251 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-2zbzn" Jul 21 02:15:14.085557 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:15:14.085529 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pwzb2" Jul 21 02:15:15.092823 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:15:15.092796 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9x9k7" Jul 21 02:16:03.986832 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:03.986800 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2jh8d"] Jul 21 02:16:03.989329 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:03.989313 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2jh8d" Jul 21 02:16:03.995184 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:03.995163 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 21 02:16:03.996384 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:03.996368 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-6nxdj\"" Jul 21 02:16:03.996462 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:03.996370 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 21 02:16:04.015714 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:04.015690 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2jh8d"] Jul 21 02:16:04.083446 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:04.083415 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm9pc\" (UniqueName: \"kubernetes.io/projected/09ff76c2-8dbc-428d-80e7-1c9a1bab1364-kube-api-access-lm9pc\") pod \"limitador-operator-controller-manager-859d7b78bd-2jh8d\" (UID: \"09ff76c2-8dbc-428d-80e7-1c9a1bab1364\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2jh8d" Jul 21 02:16:04.184161 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:04.184125 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm9pc\" (UniqueName: \"kubernetes.io/projected/09ff76c2-8dbc-428d-80e7-1c9a1bab1364-kube-api-access-lm9pc\") pod \"limitador-operator-controller-manager-859d7b78bd-2jh8d\" (UID: \"09ff76c2-8dbc-428d-80e7-1c9a1bab1364\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2jh8d" Jul 21 02:16:04.197733 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:04.197710 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm9pc\" (UniqueName: \"kubernetes.io/projected/09ff76c2-8dbc-428d-80e7-1c9a1bab1364-kube-api-access-lm9pc\") pod \"limitador-operator-controller-manager-859d7b78bd-2jh8d\" (UID: \"09ff76c2-8dbc-428d-80e7-1c9a1bab1364\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2jh8d" Jul 21 02:16:04.298815 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:04.298737 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2jh8d" Jul 21 02:16:04.458490 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:04.456650 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2jh8d"] Jul 21 02:16:05.442200 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:05.442162 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2jh8d" event={"ID":"09ff76c2-8dbc-428d-80e7-1c9a1bab1364","Type":"ContainerStarted","Data":"281439e53251d8da3ad2639a1dfaab112de4068d61ed6346f8f9f38f4519e003"} Jul 21 02:16:07.454794 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:07.454760 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2jh8d" event={"ID":"09ff76c2-8dbc-428d-80e7-1c9a1bab1364","Type":"ContainerStarted","Data":"6bfa97007a20debc2b89a24f19bc683cbadc27dc97a2c332c0311bc92ebfcbe3"} Jul 21 02:16:07.455247 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:07.454875 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2jh8d" Jul 21 02:16:07.517631 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:07.517577 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2jh8d" podStartSLOduration=2.571221431 podStartE2EDuration="4.517561753s" totalImagesPullingTime="1.946340322s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 02:16:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:16:07.516763944 +0000 UTC m=+401.536257847" watchObservedRunningTime="2026-07-21 02:16:07.517561753 +0000 UTC m=+401.537055656" Jul 21 02:16:10.425079 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:10.425046 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-4757q"] Jul 21 02:16:10.427630 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:10.427609 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-4757q" Jul 21 02:16:10.443257 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:10.443236 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-2sctp\"" Jul 21 02:16:10.444293 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:10.444258 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 21 02:16:10.445448 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:10.445426 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-4757q"] Jul 21 02:16:10.539456 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:10.539412 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2cb4\" (UniqueName: \"kubernetes.io/projected/28d4228c-c846-4f2d-b2e3-6d1c4b2298bd-kube-api-access-m2cb4\") pod \"dns-operator-controller-manager-864dd78c4c-4757q\" (UID: \"28d4228c-c846-4f2d-b2e3-6d1c4b2298bd\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-4757q" Jul 21 02:16:10.640363 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:10.640330 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2cb4\" (UniqueName: \"kubernetes.io/projected/28d4228c-c846-4f2d-b2e3-6d1c4b2298bd-kube-api-access-m2cb4\") pod \"dns-operator-controller-manager-864dd78c4c-4757q\" (UID: \"28d4228c-c846-4f2d-b2e3-6d1c4b2298bd\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-4757q" Jul 21 02:16:10.652196 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:10.652170 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2cb4\" (UniqueName: \"kubernetes.io/projected/28d4228c-c846-4f2d-b2e3-6d1c4b2298bd-kube-api-access-m2cb4\") pod \"dns-operator-controller-manager-864dd78c4c-4757q\" (UID: \"28d4228c-c846-4f2d-b2e3-6d1c4b2298bd\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-4757q" Jul 21 02:16:10.740834 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:10.740809 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-4757q" Jul 21 02:16:10.892759 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:10.892738 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-4757q"] Jul 21 02:16:10.894968 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:16:10.894945 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28d4228c_c846_4f2d_b2e3_6d1c4b2298bd.slice/crio-ea5f79991c60e6b1fea88096ee66df3044a4017815248f0cc4e8bb14512daa41 WatchSource:0}: Error finding container ea5f79991c60e6b1fea88096ee66df3044a4017815248f0cc4e8bb14512daa41: Status 404 returned error can't find the container with id ea5f79991c60e6b1fea88096ee66df3044a4017815248f0cc4e8bb14512daa41 Jul 21 02:16:11.474191 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:11.474159 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-4757q" event={"ID":"28d4228c-c846-4f2d-b2e3-6d1c4b2298bd","Type":"ContainerStarted","Data":"ea5f79991c60e6b1fea88096ee66df3044a4017815248f0cc4e8bb14512daa41"} Jul 21 02:16:13.484235 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:13.484204 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-4757q" event={"ID":"28d4228c-c846-4f2d-b2e3-6d1c4b2298bd","Type":"ContainerStarted","Data":"07f06af2e3149b1eb3e15f3a535430aaf1afb261f87a5b79e90d376851724ae2"} Jul 21 02:16:13.484634 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:13.484388 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-4757q" Jul 21 02:16:13.508675 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:13.508631 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-4757q" podStartSLOduration=1.353243904 podStartE2EDuration="3.508616984s" totalImagesPullingTime="2.15537308s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 02:16:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:16:13.506741973 +0000 UTC m=+407.526235888" watchObservedRunningTime="2026-07-21 02:16:13.508616984 +0000 UTC m=+407.528110887" Jul 21 02:16:18.460483 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:18.460452 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2jh8d" Jul 21 02:16:22.137542 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:22.137502 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4g5rc"] Jul 21 02:16:22.140315 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:22.140296 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4g5rc" Jul 21 02:16:22.143423 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:22.143403 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-h8n22\"" Jul 21 02:16:22.153936 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:22.153912 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4g5rc"] Jul 21 02:16:22.227024 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:22.226990 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4g5rc"] Jul 21 02:16:22.227322 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:16:22.227297 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-w8bm7], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4g5rc" podUID="cd8da876-3722-4672-af5b-c4271197c335" Jul 21 02:16:22.239754 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:22.239713 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cd8da876-3722-4672-af5b-c4271197c335-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-4g5rc\" (UID: \"cd8da876-3722-4672-af5b-c4271197c335\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4g5rc" Jul 21 02:16:22.240055 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:22.240025 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8bm7\" (UniqueName: \"kubernetes.io/projected/cd8da876-3722-4672-af5b-c4271197c335-kube-api-access-w8bm7\") pod \"kuadrant-operator-controller-manager-86878c769f-4g5rc\" (UID: \"cd8da876-3722-4672-af5b-c4271197c335\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4g5rc" Jul 21 02:16:22.240633 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:22.240609 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4g5rc"] Jul 21 02:16:22.260626 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:22.260589 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2jh8d"] Jul 21 02:16:22.260830 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:22.260808 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2jh8d" podUID="09ff76c2-8dbc-428d-80e7-1c9a1bab1364" containerName="manager" containerID="cri-o://6bfa97007a20debc2b89a24f19bc683cbadc27dc97a2c332c0311bc92ebfcbe3" gracePeriod=2 Jul 21 02:16:22.281966 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:22.281940 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hp5wh"] Jul 21 02:16:22.285135 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:22.285117 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hp5wh" Jul 21 02:16:22.290719 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:22.290693 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2jh8d"] Jul 21 02:16:22.317497 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:16:22.317458 2574 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"09ff76c2-8dbc-428d-80e7-1c9a1bab1364\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-859d7b78bd-2jh8d\": pods \"limitador-operator-controller-manager-859d7b78bd-2jh8d\" not found" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2jh8d" Jul 21 02:16:22.321848 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:22.321818 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hp5wh"] Jul 21 02:16:22.341529 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:22.341503 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8bm7\" (UniqueName: \"kubernetes.io/projected/cd8da876-3722-4672-af5b-c4271197c335-kube-api-access-w8bm7\") pod \"kuadrant-operator-controller-manager-86878c769f-4g5rc\" (UID: \"cd8da876-3722-4672-af5b-c4271197c335\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4g5rc" Jul 21 02:16:22.341701 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:22.341625 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cd8da876-3722-4672-af5b-c4271197c335-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-4g5rc\" (UID: \"cd8da876-3722-4672-af5b-c4271197c335\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4g5rc" Jul 21 02:16:22.341984 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:22.341964 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cd8da876-3722-4672-af5b-c4271197c335-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-4g5rc\" (UID: \"cd8da876-3722-4672-af5b-c4271197c335\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4g5rc" Jul 21 02:16:22.349523 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:16:22.349502 2574 projected.go:196] Error preparing data for projected volume kube-api-access-w8bm7 for pod kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4g5rc: failed to fetch token: pod "kuadrant-operator-controller-manager-86878c769f-4g5rc" not found Jul 21 02:16:22.349620 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:16:22.349563 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd8da876-3722-4672-af5b-c4271197c335-kube-api-access-w8bm7 podName:cd8da876-3722-4672-af5b-c4271197c335 nodeName:}" failed. No retries permitted until 2026-07-21 02:16:22.849547152 +0000 UTC m=+416.869041034 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-w8bm7" (UniqueName: "kubernetes.io/projected/cd8da876-3722-4672-af5b-c4271197c335-kube-api-access-w8bm7") pod "kuadrant-operator-controller-manager-86878c769f-4g5rc" (UID: "cd8da876-3722-4672-af5b-c4271197c335") : failed to fetch token: pod "kuadrant-operator-controller-manager-86878c769f-4g5rc" not found Jul 21 02:16:22.377867 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:16:22.377834 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-2jh8d\" is forbidden: User \"system:node:ip-10-0-141-37.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-37.ec2.internal' and this object" podUID="09ff76c2-8dbc-428d-80e7-1c9a1bab1364" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2jh8d" Jul 21 02:16:22.442432 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:22.442407 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl7nk\" (UniqueName: \"kubernetes.io/projected/59a1b2d9-71d2-43fd-bbc1-a8936442e3d6-kube-api-access-sl7nk\") pod \"kuadrant-operator-controller-manager-86878c769f-hp5wh\" (UID: \"59a1b2d9-71d2-43fd-bbc1-a8936442e3d6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hp5wh" Jul 21 02:16:22.442585 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:22.442453 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/59a1b2d9-71d2-43fd-bbc1-a8936442e3d6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-hp5wh\" (UID: \"59a1b2d9-71d2-43fd-bbc1-a8936442e3d6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hp5wh" Jul 21 02:16:22.520816 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:22.520790 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2jh8d" Jul 21 02:16:22.525548 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:22.525519 2574 generic.go:356] "Generic (PLEG): container finished" podID="09ff76c2-8dbc-428d-80e7-1c9a1bab1364" containerID="6bfa97007a20debc2b89a24f19bc683cbadc27dc97a2c332c0311bc92ebfcbe3" exitCode=0 Jul 21 02:16:22.525666 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:22.525551 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2jh8d" Jul 21 02:16:22.525666 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:22.525627 2574 scope.go:117] "RemoveContainer" containerID="6bfa97007a20debc2b89a24f19bc683cbadc27dc97a2c332c0311bc92ebfcbe3" Jul 21 02:16:22.525776 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:22.525678 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4g5rc" Jul 21 02:16:22.531676 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:22.531653 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4g5rc" Jul 21 02:16:22.533668 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:22.533652 2574 scope.go:117] "RemoveContainer" containerID="6bfa97007a20debc2b89a24f19bc683cbadc27dc97a2c332c0311bc92ebfcbe3" Jul 21 02:16:22.533947 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:16:22.533928 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bfa97007a20debc2b89a24f19bc683cbadc27dc97a2c332c0311bc92ebfcbe3\": container with ID starting with 6bfa97007a20debc2b89a24f19bc683cbadc27dc97a2c332c0311bc92ebfcbe3 not found: ID does not exist" containerID="6bfa97007a20debc2b89a24f19bc683cbadc27dc97a2c332c0311bc92ebfcbe3" Jul 21 02:16:22.533997 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:22.533957 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bfa97007a20debc2b89a24f19bc683cbadc27dc97a2c332c0311bc92ebfcbe3"} err="failed to get container status \"6bfa97007a20debc2b89a24f19bc683cbadc27dc97a2c332c0311bc92ebfcbe3\": rpc error: code = NotFound desc = could not find container \"6bfa97007a20debc2b89a24f19bc683cbadc27dc97a2c332c0311bc92ebfcbe3\": container with ID starting with 6bfa97007a20debc2b89a24f19bc683cbadc27dc97a2c332c0311bc92ebfcbe3 not found: ID does not exist" Jul 21 02:16:22.536290 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:16:22.536245 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-4g5rc\" is forbidden: User \"system:node:ip-10-0-141-37.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-37.ec2.internal' and this object" podUID="cd8da876-3722-4672-af5b-c4271197c335" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4g5rc" Jul 21 02:16:22.543617 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:22.543598 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl7nk\" (UniqueName: \"kubernetes.io/projected/59a1b2d9-71d2-43fd-bbc1-a8936442e3d6-kube-api-access-sl7nk\") pod \"kuadrant-operator-controller-manager-86878c769f-hp5wh\" (UID: \"59a1b2d9-71d2-43fd-bbc1-a8936442e3d6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hp5wh" Jul 21 02:16:22.543687 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:22.543642 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/59a1b2d9-71d2-43fd-bbc1-a8936442e3d6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-hp5wh\" (UID: \"59a1b2d9-71d2-43fd-bbc1-a8936442e3d6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hp5wh" Jul 21 02:16:22.543962 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:22.543947 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/59a1b2d9-71d2-43fd-bbc1-a8936442e3d6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-hp5wh\" (UID: \"59a1b2d9-71d2-43fd-bbc1-a8936442e3d6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hp5wh" Jul 21 02:16:22.559147 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:22.559130 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl7nk\" (UniqueName: \"kubernetes.io/projected/59a1b2d9-71d2-43fd-bbc1-a8936442e3d6-kube-api-access-sl7nk\") pod \"kuadrant-operator-controller-manager-86878c769f-hp5wh\" (UID: \"59a1b2d9-71d2-43fd-bbc1-a8936442e3d6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hp5wh" Jul 21 02:16:22.644713 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:22.644685 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cd8da876-3722-4672-af5b-c4271197c335-extensions-socket-volume\") pod \"cd8da876-3722-4672-af5b-c4271197c335\" (UID: \"cd8da876-3722-4672-af5b-c4271197c335\") " Jul 21 02:16:22.644858 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:22.644730 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lm9pc\" (UniqueName: \"kubernetes.io/projected/09ff76c2-8dbc-428d-80e7-1c9a1bab1364-kube-api-access-lm9pc\") pod \"09ff76c2-8dbc-428d-80e7-1c9a1bab1364\" (UID: \"09ff76c2-8dbc-428d-80e7-1c9a1bab1364\") " Jul 21 02:16:22.644940 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:22.644919 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd8da876-3722-4672-af5b-c4271197c335-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "cd8da876-3722-4672-af5b-c4271197c335" (UID: "cd8da876-3722-4672-af5b-c4271197c335"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 02:16:22.645027 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:22.645011 2574 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cd8da876-3722-4672-af5b-c4271197c335-extensions-socket-volume\") on node \"ip-10-0-141-37.ec2.internal\" DevicePath \"\"" Jul 21 02:16:22.645065 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:22.645035 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8bm7\" (UniqueName: \"kubernetes.io/projected/cd8da876-3722-4672-af5b-c4271197c335-kube-api-access-w8bm7\") on node \"ip-10-0-141-37.ec2.internal\" DevicePath \"\"" Jul 21 02:16:22.646832 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:22.646816 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09ff76c2-8dbc-428d-80e7-1c9a1bab1364-kube-api-access-lm9pc" (OuterVolumeSpecName: "kube-api-access-lm9pc") pod "09ff76c2-8dbc-428d-80e7-1c9a1bab1364" (UID: "09ff76c2-8dbc-428d-80e7-1c9a1bab1364"). InnerVolumeSpecName "kube-api-access-lm9pc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 02:16:22.647948 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:22.647930 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hp5wh" Jul 21 02:16:22.745974 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:22.745939 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm9pc\" (UniqueName: \"kubernetes.io/projected/09ff76c2-8dbc-428d-80e7-1c9a1bab1364-kube-api-access-lm9pc\") on node \"ip-10-0-141-37.ec2.internal\" DevicePath \"\"" Jul 21 02:16:22.806000 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:22.805971 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hp5wh"] Jul 21 02:16:22.808603 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:16:22.808580 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59a1b2d9_71d2_43fd_bbc1_a8936442e3d6.slice/crio-0bb18b7e77c4434da9ebd9e4d6b2fa7884219a345c18ace17ec1fe3c17ad1f6e WatchSource:0}: Error finding container 0bb18b7e77c4434da9ebd9e4d6b2fa7884219a345c18ace17ec1fe3c17ad1f6e: Status 404 returned error can't find the container with id 0bb18b7e77c4434da9ebd9e4d6b2fa7884219a345c18ace17ec1fe3c17ad1f6e Jul 21 02:16:22.836940 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:16:22.836911 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-4g5rc\" is forbidden: User \"system:node:ip-10-0-141-37.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-37.ec2.internal' and this object" podUID="cd8da876-3722-4672-af5b-c4271197c335" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4g5rc" Jul 21 02:16:23.534764 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:23.534729 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hp5wh" event={"ID":"59a1b2d9-71d2-43fd-bbc1-a8936442e3d6","Type":"ContainerStarted","Data":"0bb18b7e77c4434da9ebd9e4d6b2fa7884219a345c18ace17ec1fe3c17ad1f6e"} Jul 21 02:16:23.534764 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:23.534756 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4g5rc" Jul 21 02:16:23.537562 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:16:23.537530 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-4g5rc\" is forbidden: User \"system:node:ip-10-0-141-37.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-37.ec2.internal' and this object" podUID="cd8da876-3722-4672-af5b-c4271197c335" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4g5rc" Jul 21 02:16:23.546367 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:16:23.546331 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-4g5rc\" is forbidden: User \"system:node:ip-10-0-141-37.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-37.ec2.internal' and this object" podUID="cd8da876-3722-4672-af5b-c4271197c335" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4g5rc" Jul 21 02:16:24.485121 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:24.485091 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09ff76c2-8dbc-428d-80e7-1c9a1bab1364" path="/var/lib/kubelet/pods/09ff76c2-8dbc-428d-80e7-1c9a1bab1364/volumes" Jul 21 02:16:24.485433 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:24.485419 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd8da876-3722-4672-af5b-c4271197c335" path="/var/lib/kubelet/pods/cd8da876-3722-4672-af5b-c4271197c335/volumes" Jul 21 02:16:24.490106 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:24.490083 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-4757q" Jul 21 02:16:27.557144 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:27.557106 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hp5wh" event={"ID":"59a1b2d9-71d2-43fd-bbc1-a8936442e3d6","Type":"ContainerStarted","Data":"3c2145dd5c8682245f19025b11a5303ea992b03426e262d65da2d300f0964b7d"} Jul 21 02:16:27.557630 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:27.557368 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hp5wh" Jul 21 02:16:27.587120 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:27.587070 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hp5wh" podStartSLOduration=1.24021731 podStartE2EDuration="5.587054608s" totalImagesPullingTime="4.346837298s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 02:16:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:16:27.584879136 +0000 UTC m=+421.604373067" watchObservedRunningTime="2026-07-21 02:16:27.587054608 +0000 UTC m=+421.606548511" Jul 21 02:16:38.563969 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:38.563939 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hp5wh" Jul 21 02:16:56.627656 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:56.627581 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-vtgdj"] Jul 21 02:16:56.628066 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:56.627926 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09ff76c2-8dbc-428d-80e7-1c9a1bab1364" containerName="manager" Jul 21 02:16:56.628066 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:56.627937 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="09ff76c2-8dbc-428d-80e7-1c9a1bab1364" containerName="manager" Jul 21 02:16:56.628066 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:56.628005 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="09ff76c2-8dbc-428d-80e7-1c9a1bab1364" containerName="manager" Jul 21 02:16:56.631119 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:56.631102 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-vtgdj" Jul 21 02:16:56.633716 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:56.633694 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 21 02:16:56.633814 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:56.633782 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-kvkt4\"" Jul 21 02:16:56.638619 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:56.638412 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-vtgdj"] Jul 21 02:16:56.721545 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:56.721514 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-vtgdj"] Jul 21 02:16:56.751363 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:56.751333 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5n24\" (UniqueName: \"kubernetes.io/projected/956c26db-268c-407a-ac7e-e988bacd54a1-kube-api-access-s5n24\") pod \"limitador-limitador-64c94fd64f-vtgdj\" (UID: \"956c26db-268c-407a-ac7e-e988bacd54a1\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-vtgdj" Jul 21 02:16:56.751503 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:56.751453 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/956c26db-268c-407a-ac7e-e988bacd54a1-config-file\") pod \"limitador-limitador-64c94fd64f-vtgdj\" (UID: \"956c26db-268c-407a-ac7e-e988bacd54a1\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-vtgdj" Jul 21 02:16:56.851939 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:56.851912 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5n24\" (UniqueName: \"kubernetes.io/projected/956c26db-268c-407a-ac7e-e988bacd54a1-kube-api-access-s5n24\") pod \"limitador-limitador-64c94fd64f-vtgdj\" (UID: \"956c26db-268c-407a-ac7e-e988bacd54a1\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-vtgdj" Jul 21 02:16:56.852105 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:56.852010 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/956c26db-268c-407a-ac7e-e988bacd54a1-config-file\") pod \"limitador-limitador-64c94fd64f-vtgdj\" (UID: \"956c26db-268c-407a-ac7e-e988bacd54a1\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-vtgdj" Jul 21 02:16:56.852691 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:56.852670 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/956c26db-268c-407a-ac7e-e988bacd54a1-config-file\") pod \"limitador-limitador-64c94fd64f-vtgdj\" (UID: \"956c26db-268c-407a-ac7e-e988bacd54a1\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-vtgdj" Jul 21 02:16:56.860460 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:56.860437 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5n24\" (UniqueName: \"kubernetes.io/projected/956c26db-268c-407a-ac7e-e988bacd54a1-kube-api-access-s5n24\") pod \"limitador-limitador-64c94fd64f-vtgdj\" (UID: \"956c26db-268c-407a-ac7e-e988bacd54a1\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-vtgdj" Jul 21 02:16:56.942472 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:56.942450 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-vtgdj" Jul 21 02:16:57.088999 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:57.088942 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-vtgdj"] Jul 21 02:16:57.091711 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:16:57.091681 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod956c26db_268c_407a_ac7e_e988bacd54a1.slice/crio-b8e27346e3d1000b1c89414c475616b9d1aa093ba25fd93fa5b3fd35088ffa92 WatchSource:0}: Error finding container b8e27346e3d1000b1c89414c475616b9d1aa093ba25fd93fa5b3fd35088ffa92: Status 404 returned error can't find the container with id b8e27346e3d1000b1c89414c475616b9d1aa093ba25fd93fa5b3fd35088ffa92 Jul 21 02:16:57.695253 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:16:57.695220 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-vtgdj" event={"ID":"956c26db-268c-407a-ac7e-e988bacd54a1","Type":"ContainerStarted","Data":"b8e27346e3d1000b1c89414c475616b9d1aa093ba25fd93fa5b3fd35088ffa92"} Jul 21 02:17:00.713899 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:00.713857 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-vtgdj" event={"ID":"956c26db-268c-407a-ac7e-e988bacd54a1","Type":"ContainerStarted","Data":"f6e2214cbf0c29c284586ca6a9bfa8891ab657ba7e78f6dd0194ec5a5888e741"} Jul 21 02:17:00.714299 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:00.713974 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-vtgdj" Jul 21 02:17:00.732848 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:00.732796 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-vtgdj" podStartSLOduration=2.054593046 podStartE2EDuration="4.732779554s" totalImagesPullingTime="2.678186508s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 02:16:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:17:00.7305707 +0000 UTC m=+454.750064604" watchObservedRunningTime="2026-07-21 02:17:00.732779554 +0000 UTC m=+454.752273457" Jul 21 02:17:11.720395 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:11.720366 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-vtgdj" Jul 21 02:17:13.643280 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:13.643240 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-vtgdj"] Jul 21 02:17:13.643673 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:13.643450 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-vtgdj" podUID="956c26db-268c-407a-ac7e-e988bacd54a1" containerName="limitador" containerID="cri-o://f6e2214cbf0c29c284586ca6a9bfa8891ab657ba7e78f6dd0194ec5a5888e741" gracePeriod=30 Jul 21 02:17:14.193545 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:14.193523 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-vtgdj" Jul 21 02:17:14.288494 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:14.288434 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/956c26db-268c-407a-ac7e-e988bacd54a1-config-file\") pod \"956c26db-268c-407a-ac7e-e988bacd54a1\" (UID: \"956c26db-268c-407a-ac7e-e988bacd54a1\") " Jul 21 02:17:14.288641 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:14.288494 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s5n24\" (UniqueName: \"kubernetes.io/projected/956c26db-268c-407a-ac7e-e988bacd54a1-kube-api-access-s5n24\") pod \"956c26db-268c-407a-ac7e-e988bacd54a1\" (UID: \"956c26db-268c-407a-ac7e-e988bacd54a1\") " Jul 21 02:17:14.288765 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:14.288742 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/956c26db-268c-407a-ac7e-e988bacd54a1-config-file" (OuterVolumeSpecName: "config-file") pod "956c26db-268c-407a-ac7e-e988bacd54a1" (UID: "956c26db-268c-407a-ac7e-e988bacd54a1"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 02:17:14.290537 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:14.290504 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/956c26db-268c-407a-ac7e-e988bacd54a1-kube-api-access-s5n24" (OuterVolumeSpecName: "kube-api-access-s5n24") pod "956c26db-268c-407a-ac7e-e988bacd54a1" (UID: "956c26db-268c-407a-ac7e-e988bacd54a1"). InnerVolumeSpecName "kube-api-access-s5n24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 02:17:14.389054 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:14.389024 2574 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/956c26db-268c-407a-ac7e-e988bacd54a1-config-file\") on node \"ip-10-0-141-37.ec2.internal\" DevicePath \"\"" Jul 21 02:17:14.389054 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:14.389050 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5n24\" (UniqueName: \"kubernetes.io/projected/956c26db-268c-407a-ac7e-e988bacd54a1-kube-api-access-s5n24\") on node \"ip-10-0-141-37.ec2.internal\" DevicePath \"\"" Jul 21 02:17:14.776028 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:14.775995 2574 generic.go:356] "Generic (PLEG): container finished" podID="956c26db-268c-407a-ac7e-e988bacd54a1" containerID="f6e2214cbf0c29c284586ca6a9bfa8891ab657ba7e78f6dd0194ec5a5888e741" exitCode=0 Jul 21 02:17:14.776498 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:14.776043 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-vtgdj" Jul 21 02:17:14.776498 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:14.776084 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-vtgdj" event={"ID":"956c26db-268c-407a-ac7e-e988bacd54a1","Type":"ContainerDied","Data":"f6e2214cbf0c29c284586ca6a9bfa8891ab657ba7e78f6dd0194ec5a5888e741"} Jul 21 02:17:14.776498 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:14.776142 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-vtgdj" event={"ID":"956c26db-268c-407a-ac7e-e988bacd54a1","Type":"ContainerDied","Data":"b8e27346e3d1000b1c89414c475616b9d1aa093ba25fd93fa5b3fd35088ffa92"} Jul 21 02:17:14.776498 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:14.776176 2574 scope.go:117] "RemoveContainer" containerID="f6e2214cbf0c29c284586ca6a9bfa8891ab657ba7e78f6dd0194ec5a5888e741" Jul 21 02:17:14.784463 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:14.784444 2574 scope.go:117] "RemoveContainer" containerID="f6e2214cbf0c29c284586ca6a9bfa8891ab657ba7e78f6dd0194ec5a5888e741" Jul 21 02:17:14.784709 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:17:14.784689 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6e2214cbf0c29c284586ca6a9bfa8891ab657ba7e78f6dd0194ec5a5888e741\": container with ID starting with f6e2214cbf0c29c284586ca6a9bfa8891ab657ba7e78f6dd0194ec5a5888e741 not found: ID does not exist" containerID="f6e2214cbf0c29c284586ca6a9bfa8891ab657ba7e78f6dd0194ec5a5888e741" Jul 21 02:17:14.784763 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:14.784721 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6e2214cbf0c29c284586ca6a9bfa8891ab657ba7e78f6dd0194ec5a5888e741"} err="failed to get container status \"f6e2214cbf0c29c284586ca6a9bfa8891ab657ba7e78f6dd0194ec5a5888e741\": rpc error: code = NotFound desc = could not find container \"f6e2214cbf0c29c284586ca6a9bfa8891ab657ba7e78f6dd0194ec5a5888e741\": container with ID starting with f6e2214cbf0c29c284586ca6a9bfa8891ab657ba7e78f6dd0194ec5a5888e741 not found: ID does not exist" Jul 21 02:17:14.792413 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:14.792394 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-vtgdj"] Jul 21 02:17:14.796675 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:14.796656 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-vtgdj"] Jul 21 02:17:16.484950 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:16.484920 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="956c26db-268c-407a-ac7e-e988bacd54a1" path="/var/lib/kubelet/pods/956c26db-268c-407a-ac7e-e988bacd54a1/volumes" Jul 21 02:17:18.326139 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:18.326110 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-4bprz"] Jul 21 02:17:18.326514 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:18.326485 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="956c26db-268c-407a-ac7e-e988bacd54a1" containerName="limitador" Jul 21 02:17:18.326514 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:18.326497 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="956c26db-268c-407a-ac7e-e988bacd54a1" containerName="limitador" Jul 21 02:17:18.326591 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:18.326580 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="956c26db-268c-407a-ac7e-e988bacd54a1" containerName="limitador" Jul 21 02:17:18.331161 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:18.331137 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4bprz" Jul 21 02:17:18.334436 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:18.334415 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 21 02:17:18.334787 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:18.334771 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-h6qs8\"" Jul 21 02:17:18.334887 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:18.334782 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 21 02:17:18.334887 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:18.334840 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 21 02:17:18.339821 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:18.339800 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-4bprz"] Jul 21 02:17:18.427436 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:18.427408 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/db59b237-fdb3-4c81-86d2-1cfd7fa4cf4a-data\") pod \"postgres-69fb4594cf-4bprz\" (UID: \"db59b237-fdb3-4c81-86d2-1cfd7fa4cf4a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4bprz" Jul 21 02:17:18.427575 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:18.427463 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-985bd\" (UniqueName: \"kubernetes.io/projected/db59b237-fdb3-4c81-86d2-1cfd7fa4cf4a-kube-api-access-985bd\") pod \"postgres-69fb4594cf-4bprz\" (UID: \"db59b237-fdb3-4c81-86d2-1cfd7fa4cf4a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4bprz" Jul 21 02:17:18.528351 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:18.528324 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/db59b237-fdb3-4c81-86d2-1cfd7fa4cf4a-data\") pod \"postgres-69fb4594cf-4bprz\" (UID: \"db59b237-fdb3-4c81-86d2-1cfd7fa4cf4a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4bprz" Jul 21 02:17:18.528482 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:18.528370 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-985bd\" (UniqueName: \"kubernetes.io/projected/db59b237-fdb3-4c81-86d2-1cfd7fa4cf4a-kube-api-access-985bd\") pod \"postgres-69fb4594cf-4bprz\" (UID: \"db59b237-fdb3-4c81-86d2-1cfd7fa4cf4a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4bprz" Jul 21 02:17:18.528659 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:18.528643 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/db59b237-fdb3-4c81-86d2-1cfd7fa4cf4a-data\") pod \"postgres-69fb4594cf-4bprz\" (UID: \"db59b237-fdb3-4c81-86d2-1cfd7fa4cf4a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4bprz" Jul 21 02:17:18.537485 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:18.537459 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-985bd\" (UniqueName: \"kubernetes.io/projected/db59b237-fdb3-4c81-86d2-1cfd7fa4cf4a-kube-api-access-985bd\") pod \"postgres-69fb4594cf-4bprz\" (UID: \"db59b237-fdb3-4c81-86d2-1cfd7fa4cf4a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4bprz" Jul 21 02:17:18.642222 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:18.642167 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4bprz" Jul 21 02:17:18.768283 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:18.768238 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-4bprz"] Jul 21 02:17:18.771884 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:17:18.771854 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb59b237_fdb3_4c81_86d2_1cfd7fa4cf4a.slice/crio-a1405aa6429b6d961af3061c3c27fd0eaba76968ffae6b0088d444263edc4cc9 WatchSource:0}: Error finding container a1405aa6429b6d961af3061c3c27fd0eaba76968ffae6b0088d444263edc4cc9: Status 404 returned error can't find the container with id a1405aa6429b6d961af3061c3c27fd0eaba76968ffae6b0088d444263edc4cc9 Jul 21 02:17:18.794303 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:18.794252 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4bprz" event={"ID":"db59b237-fdb3-4c81-86d2-1cfd7fa4cf4a","Type":"ContainerStarted","Data":"a1405aa6429b6d961af3061c3c27fd0eaba76968ffae6b0088d444263edc4cc9"} Jul 21 02:17:25.833596 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:25.833503 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4bprz" event={"ID":"db59b237-fdb3-4c81-86d2-1cfd7fa4cf4a","Type":"ContainerStarted","Data":"af60743393ef88e9beb5c869f09f5c496f0862afa4891913af730fcb035cf944"} Jul 21 02:17:25.833956 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:25.833643 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4bprz" Jul 21 02:17:25.853374 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:25.853324 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4bprz" podStartSLOduration=1.149076207 podStartE2EDuration="7.853308523s" totalImagesPullingTime="6.704232316s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 02:17:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:17:25.850056454 +0000 UTC m=+479.869550380" watchObservedRunningTime="2026-07-21 02:17:25.853308523 +0000 UTC m=+479.872802427" Jul 21 02:17:31.865686 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:31.865657 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4bprz" Jul 21 02:17:40.333902 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:40.333867 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-577758d896-2m6k9"] Jul 21 02:17:40.341913 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:40.341889 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577758d896-2m6k9" Jul 21 02:17:40.345017 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:40.344991 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 21 02:17:40.345147 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:40.345030 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 21 02:17:40.346018 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:40.346003 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-85mqn\"" Jul 21 02:17:40.363552 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:40.363521 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-577758d896-2m6k9"] Jul 21 02:17:40.410383 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:40.410341 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmpjt\" (UniqueName: \"kubernetes.io/projected/a0801590-3898-4051-98a3-357ca5527c90-kube-api-access-xmpjt\") pod \"maas-controller-577758d896-2m6k9\" (UID: \"a0801590-3898-4051-98a3-357ca5527c90\") " pod="opendatahub/maas-controller-577758d896-2m6k9" Jul 21 02:17:40.410566 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:40.410389 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a0801590-3898-4051-98a3-357ca5527c90-webhook-cert\") pod \"maas-controller-577758d896-2m6k9\" (UID: \"a0801590-3898-4051-98a3-357ca5527c90\") " pod="opendatahub/maas-controller-577758d896-2m6k9" Jul 21 02:17:40.484925 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:40.484900 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-65f68477d7-rtzbp"] Jul 21 02:17:40.488554 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:40.488535 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65f68477d7-rtzbp" Jul 21 02:17:40.498237 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:40.498209 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65f68477d7-rtzbp"] Jul 21 02:17:40.511279 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:40.511246 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmpjt\" (UniqueName: \"kubernetes.io/projected/a0801590-3898-4051-98a3-357ca5527c90-kube-api-access-xmpjt\") pod \"maas-controller-577758d896-2m6k9\" (UID: \"a0801590-3898-4051-98a3-357ca5527c90\") " pod="opendatahub/maas-controller-577758d896-2m6k9" Jul 21 02:17:40.511466 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:40.511304 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwk4c\" (UniqueName: \"kubernetes.io/projected/9511c95c-704b-49d4-abb6-5db8cb838e08-kube-api-access-fwk4c\") pod \"maas-controller-65f68477d7-rtzbp\" (UID: \"9511c95c-704b-49d4-abb6-5db8cb838e08\") " pod="opendatahub/maas-controller-65f68477d7-rtzbp" Jul 21 02:17:40.511466 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:40.511339 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a0801590-3898-4051-98a3-357ca5527c90-webhook-cert\") pod \"maas-controller-577758d896-2m6k9\" (UID: \"a0801590-3898-4051-98a3-357ca5527c90\") " pod="opendatahub/maas-controller-577758d896-2m6k9" Jul 21 02:17:40.511588 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:40.511470 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9511c95c-704b-49d4-abb6-5db8cb838e08-webhook-cert\") pod \"maas-controller-65f68477d7-rtzbp\" (UID: \"9511c95c-704b-49d4-abb6-5db8cb838e08\") " pod="opendatahub/maas-controller-65f68477d7-rtzbp" Jul 21 02:17:40.513879 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:40.513856 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a0801590-3898-4051-98a3-357ca5527c90-webhook-cert\") pod \"maas-controller-577758d896-2m6k9\" (UID: \"a0801590-3898-4051-98a3-357ca5527c90\") " pod="opendatahub/maas-controller-577758d896-2m6k9" Jul 21 02:17:40.522287 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:40.522247 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmpjt\" (UniqueName: \"kubernetes.io/projected/a0801590-3898-4051-98a3-357ca5527c90-kube-api-access-xmpjt\") pod \"maas-controller-577758d896-2m6k9\" (UID: \"a0801590-3898-4051-98a3-357ca5527c90\") " pod="opendatahub/maas-controller-577758d896-2m6k9" Jul 21 02:17:40.612653 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:40.612553 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9511c95c-704b-49d4-abb6-5db8cb838e08-webhook-cert\") pod \"maas-controller-65f68477d7-rtzbp\" (UID: \"9511c95c-704b-49d4-abb6-5db8cb838e08\") " pod="opendatahub/maas-controller-65f68477d7-rtzbp" Jul 21 02:17:40.612653 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:40.612607 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwk4c\" (UniqueName: \"kubernetes.io/projected/9511c95c-704b-49d4-abb6-5db8cb838e08-kube-api-access-fwk4c\") pod \"maas-controller-65f68477d7-rtzbp\" (UID: \"9511c95c-704b-49d4-abb6-5db8cb838e08\") " pod="opendatahub/maas-controller-65f68477d7-rtzbp" Jul 21 02:17:40.615051 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:40.615024 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9511c95c-704b-49d4-abb6-5db8cb838e08-webhook-cert\") pod \"maas-controller-65f68477d7-rtzbp\" (UID: \"9511c95c-704b-49d4-abb6-5db8cb838e08\") " pod="opendatahub/maas-controller-65f68477d7-rtzbp" Jul 21 02:17:40.621532 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:40.621511 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwk4c\" (UniqueName: \"kubernetes.io/projected/9511c95c-704b-49d4-abb6-5db8cb838e08-kube-api-access-fwk4c\") pod \"maas-controller-65f68477d7-rtzbp\" (UID: \"9511c95c-704b-49d4-abb6-5db8cb838e08\") " pod="opendatahub/maas-controller-65f68477d7-rtzbp" Jul 21 02:17:40.652184 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:40.652158 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577758d896-2m6k9" Jul 21 02:17:40.788396 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:40.788373 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-577758d896-2m6k9"] Jul 21 02:17:40.790247 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:17:40.790224 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0801590_3898_4051_98a3_357ca5527c90.slice/crio-840986e783d14f555c8be693a1729b94d9b496d50a184cc9f1f03ba5127b3e00 WatchSource:0}: Error finding container 840986e783d14f555c8be693a1729b94d9b496d50a184cc9f1f03ba5127b3e00: Status 404 returned error can't find the container with id 840986e783d14f555c8be693a1729b94d9b496d50a184cc9f1f03ba5127b3e00 Jul 21 02:17:40.799732 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:40.799713 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65f68477d7-rtzbp" Jul 21 02:17:40.903810 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:40.903779 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577758d896-2m6k9" event={"ID":"a0801590-3898-4051-98a3-357ca5527c90","Type":"ContainerStarted","Data":"840986e783d14f555c8be693a1729b94d9b496d50a184cc9f1f03ba5127b3e00"} Jul 21 02:17:40.940151 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:40.940129 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65f68477d7-rtzbp"] Jul 21 02:17:40.942616 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:17:40.942590 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9511c95c_704b_49d4_abb6_5db8cb838e08.slice/crio-0e4344ebe1d1893ebf8f069721a2e7331ab8f3c6c51ff7d3c954318b8e140d11 WatchSource:0}: Error finding container 0e4344ebe1d1893ebf8f069721a2e7331ab8f3c6c51ff7d3c954318b8e140d11: Status 404 returned error can't find the container with id 0e4344ebe1d1893ebf8f069721a2e7331ab8f3c6c51ff7d3c954318b8e140d11 Jul 21 02:17:41.914429 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:41.914250 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65f68477d7-rtzbp" event={"ID":"9511c95c-704b-49d4-abb6-5db8cb838e08","Type":"ContainerStarted","Data":"0e4344ebe1d1893ebf8f069721a2e7331ab8f3c6c51ff7d3c954318b8e140d11"} Jul 21 02:17:43.928086 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:43.928053 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577758d896-2m6k9" event={"ID":"a0801590-3898-4051-98a3-357ca5527c90","Type":"ContainerStarted","Data":"32112faa960d925132aa471e5c8d42fdc8a0fda81f837d95f37ea89abaf13ec2"} Jul 21 02:17:43.928493 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:43.928185 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-577758d896-2m6k9" Jul 21 02:17:43.930087 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:43.930061 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65f68477d7-rtzbp" event={"ID":"9511c95c-704b-49d4-abb6-5db8cb838e08","Type":"ContainerStarted","Data":"7f1143780fb4614af7296dbe6e205b3f0395d0564289b7d5589fa2f60423d47c"} Jul 21 02:17:43.930205 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:43.930121 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-65f68477d7-rtzbp" Jul 21 02:17:43.964844 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:43.964791 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-577758d896-2m6k9" podStartSLOduration=1.458217837 podStartE2EDuration="3.964771285s" totalImagesPullingTime="2.506553448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 02:17:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:17:43.963011633 +0000 UTC m=+497.982505555" watchObservedRunningTime="2026-07-21 02:17:43.964771285 +0000 UTC m=+497.984265191" Jul 21 02:17:54.937733 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:54.937699 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-65f68477d7-rtzbp" Jul 21 02:17:54.938114 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:54.937858 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-577758d896-2m6k9" Jul 21 02:17:54.960048 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:54.959997 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-65f68477d7-rtzbp" podStartSLOduration=12.599753867 podStartE2EDuration="14.959985739s" totalImagesPullingTime="2.360231872s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 02:17:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:17:44.020465403 +0000 UTC m=+498.039959307" watchObservedRunningTime="2026-07-21 02:17:54.959985739 +0000 UTC m=+508.979479622" Jul 21 02:17:55.017329 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:55.017300 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-577758d896-2m6k9"] Jul 21 02:17:55.017510 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:55.017490 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-577758d896-2m6k9" podUID="a0801590-3898-4051-98a3-357ca5527c90" containerName="manager" containerID="cri-o://32112faa960d925132aa471e5c8d42fdc8a0fda81f837d95f37ea89abaf13ec2" gracePeriod=30 Jul 21 02:17:55.257618 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:55.257594 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577758d896-2m6k9" Jul 21 02:17:55.339279 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:55.339246 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a0801590-3898-4051-98a3-357ca5527c90-webhook-cert\") pod \"a0801590-3898-4051-98a3-357ca5527c90\" (UID: \"a0801590-3898-4051-98a3-357ca5527c90\") " Jul 21 02:17:55.339412 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:55.339304 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xmpjt\" (UniqueName: \"kubernetes.io/projected/a0801590-3898-4051-98a3-357ca5527c90-kube-api-access-xmpjt\") pod \"a0801590-3898-4051-98a3-357ca5527c90\" (UID: \"a0801590-3898-4051-98a3-357ca5527c90\") " Jul 21 02:17:55.341344 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:55.341313 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0801590-3898-4051-98a3-357ca5527c90-kube-api-access-xmpjt" (OuterVolumeSpecName: "kube-api-access-xmpjt") pod "a0801590-3898-4051-98a3-357ca5527c90" (UID: "a0801590-3898-4051-98a3-357ca5527c90"). InnerVolumeSpecName "kube-api-access-xmpjt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 02:17:55.341445 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:55.341357 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0801590-3898-4051-98a3-357ca5527c90-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "a0801590-3898-4051-98a3-357ca5527c90" (UID: "a0801590-3898-4051-98a3-357ca5527c90"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 02:17:55.440484 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:55.440462 2574 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a0801590-3898-4051-98a3-357ca5527c90-webhook-cert\") on node \"ip-10-0-141-37.ec2.internal\" DevicePath \"\"" Jul 21 02:17:55.440484 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:55.440483 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmpjt\" (UniqueName: \"kubernetes.io/projected/a0801590-3898-4051-98a3-357ca5527c90-kube-api-access-xmpjt\") on node \"ip-10-0-141-37.ec2.internal\" DevicePath \"\"" Jul 21 02:17:55.995378 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:55.995346 2574 generic.go:356] "Generic (PLEG): container finished" podID="a0801590-3898-4051-98a3-357ca5527c90" containerID="32112faa960d925132aa471e5c8d42fdc8a0fda81f837d95f37ea89abaf13ec2" exitCode=0 Jul 21 02:17:55.995784 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:55.995391 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577758d896-2m6k9" Jul 21 02:17:55.995784 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:55.995425 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577758d896-2m6k9" event={"ID":"a0801590-3898-4051-98a3-357ca5527c90","Type":"ContainerDied","Data":"32112faa960d925132aa471e5c8d42fdc8a0fda81f837d95f37ea89abaf13ec2"} Jul 21 02:17:55.995784 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:55.995477 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577758d896-2m6k9" event={"ID":"a0801590-3898-4051-98a3-357ca5527c90","Type":"ContainerDied","Data":"840986e783d14f555c8be693a1729b94d9b496d50a184cc9f1f03ba5127b3e00"} Jul 21 02:17:55.995784 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:55.995497 2574 scope.go:117] "RemoveContainer" containerID="32112faa960d925132aa471e5c8d42fdc8a0fda81f837d95f37ea89abaf13ec2" Jul 21 02:17:56.005007 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:56.004992 2574 scope.go:117] "RemoveContainer" containerID="32112faa960d925132aa471e5c8d42fdc8a0fda81f837d95f37ea89abaf13ec2" Jul 21 02:17:56.005300 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:17:56.005282 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32112faa960d925132aa471e5c8d42fdc8a0fda81f837d95f37ea89abaf13ec2\": container with ID starting with 32112faa960d925132aa471e5c8d42fdc8a0fda81f837d95f37ea89abaf13ec2 not found: ID does not exist" containerID="32112faa960d925132aa471e5c8d42fdc8a0fda81f837d95f37ea89abaf13ec2" Jul 21 02:17:56.005379 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:56.005304 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32112faa960d925132aa471e5c8d42fdc8a0fda81f837d95f37ea89abaf13ec2"} err="failed to get container status \"32112faa960d925132aa471e5c8d42fdc8a0fda81f837d95f37ea89abaf13ec2\": rpc error: code = NotFound desc = could not find container \"32112faa960d925132aa471e5c8d42fdc8a0fda81f837d95f37ea89abaf13ec2\": container with ID starting with 32112faa960d925132aa471e5c8d42fdc8a0fda81f837d95f37ea89abaf13ec2 not found: ID does not exist" Jul 21 02:17:56.019791 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:56.019770 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-577758d896-2m6k9"] Jul 21 02:17:56.029644 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:56.029625 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-577758d896-2m6k9"] Jul 21 02:17:56.484614 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:17:56.484582 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0801590-3898-4051-98a3-357ca5527c90" path="/var/lib/kubelet/pods/a0801590-3898-4051-98a3-357ca5527c90/volumes" Jul 21 02:18:15.943073 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:15.943032 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-55f777c84-9xkgr"] Jul 21 02:18:15.943641 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:15.943619 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0801590-3898-4051-98a3-357ca5527c90" containerName="manager" Jul 21 02:18:15.943718 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:15.943643 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0801590-3898-4051-98a3-357ca5527c90" containerName="manager" Jul 21 02:18:15.943775 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:15.943767 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0801590-3898-4051-98a3-357ca5527c90" containerName="manager" Jul 21 02:18:15.970424 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:15.970392 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55f777c84-9xkgr" Jul 21 02:18:15.972192 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:15.972169 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-55f777c84-9xkgr"] Jul 21 02:18:15.974372 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:15.974350 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 21 02:18:16.119147 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:16.119121 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32e2eb32-3083-4856-a8b5-97eeaa74175d-openshift-service-ca\") pod \"maas-api-55f777c84-9xkgr\" (UID: \"32e2eb32-3083-4856-a8b5-97eeaa74175d\") " pod="odh-ai-gateway-infra/maas-api-55f777c84-9xkgr" Jul 21 02:18:16.119343 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:16.119162 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxmdf\" (UniqueName: \"kubernetes.io/projected/32e2eb32-3083-4856-a8b5-97eeaa74175d-kube-api-access-bxmdf\") pod \"maas-api-55f777c84-9xkgr\" (UID: \"32e2eb32-3083-4856-a8b5-97eeaa74175d\") " pod="odh-ai-gateway-infra/maas-api-55f777c84-9xkgr" Jul 21 02:18:16.119343 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:16.119220 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32e2eb32-3083-4856-a8b5-97eeaa74175d-maas-api-tls\") pod \"maas-api-55f777c84-9xkgr\" (UID: \"32e2eb32-3083-4856-a8b5-97eeaa74175d\") " pod="odh-ai-gateway-infra/maas-api-55f777c84-9xkgr" Jul 21 02:18:16.220146 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:16.220075 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxmdf\" (UniqueName: \"kubernetes.io/projected/32e2eb32-3083-4856-a8b5-97eeaa74175d-kube-api-access-bxmdf\") pod \"maas-api-55f777c84-9xkgr\" (UID: \"32e2eb32-3083-4856-a8b5-97eeaa74175d\") " pod="odh-ai-gateway-infra/maas-api-55f777c84-9xkgr" Jul 21 02:18:16.220146 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:16.220127 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32e2eb32-3083-4856-a8b5-97eeaa74175d-maas-api-tls\") pod \"maas-api-55f777c84-9xkgr\" (UID: \"32e2eb32-3083-4856-a8b5-97eeaa74175d\") " pod="odh-ai-gateway-infra/maas-api-55f777c84-9xkgr" Jul 21 02:18:16.220339 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:16.220186 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32e2eb32-3083-4856-a8b5-97eeaa74175d-openshift-service-ca\") pod \"maas-api-55f777c84-9xkgr\" (UID: \"32e2eb32-3083-4856-a8b5-97eeaa74175d\") " pod="odh-ai-gateway-infra/maas-api-55f777c84-9xkgr" Jul 21 02:18:16.220881 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:16.220864 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32e2eb32-3083-4856-a8b5-97eeaa74175d-openshift-service-ca\") pod \"maas-api-55f777c84-9xkgr\" (UID: \"32e2eb32-3083-4856-a8b5-97eeaa74175d\") " pod="odh-ai-gateway-infra/maas-api-55f777c84-9xkgr" Jul 21 02:18:16.222598 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:16.222580 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32e2eb32-3083-4856-a8b5-97eeaa74175d-maas-api-tls\") pod \"maas-api-55f777c84-9xkgr\" (UID: \"32e2eb32-3083-4856-a8b5-97eeaa74175d\") " pod="odh-ai-gateway-infra/maas-api-55f777c84-9xkgr" Jul 21 02:18:16.230660 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:16.230640 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxmdf\" (UniqueName: \"kubernetes.io/projected/32e2eb32-3083-4856-a8b5-97eeaa74175d-kube-api-access-bxmdf\") pod \"maas-api-55f777c84-9xkgr\" (UID: \"32e2eb32-3083-4856-a8b5-97eeaa74175d\") " pod="odh-ai-gateway-infra/maas-api-55f777c84-9xkgr" Jul 21 02:18:16.282750 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:16.282731 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55f777c84-9xkgr" Jul 21 02:18:16.423810 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:16.423783 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-55f777c84-9xkgr"] Jul 21 02:18:16.426563 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:18:16.426533 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32e2eb32_3083_4856_a8b5_97eeaa74175d.slice/crio-a45b912728b863652fac152d1eb2f0d87f8ee5be07a1151ee6e877f1fee4a48e WatchSource:0}: Error finding container a45b912728b863652fac152d1eb2f0d87f8ee5be07a1151ee6e877f1fee4a48e: Status 404 returned error can't find the container with id a45b912728b863652fac152d1eb2f0d87f8ee5be07a1151ee6e877f1fee4a48e Jul 21 02:18:16.874596 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:16.874557 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-774v7"] Jul 21 02:18:16.879184 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:16.879164 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-774v7" Jul 21 02:18:16.881783 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:16.881767 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 21 02:18:16.893030 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:16.892999 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-774v7"] Jul 21 02:18:16.946617 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:16.946588 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-222pw"] Jul 21 02:18:16.950050 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:16.950035 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-222pw" Jul 21 02:18:16.953137 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:16.953110 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-nlq94\"" Jul 21 02:18:16.972357 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:16.972339 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-222pw"] Jul 21 02:18:17.027590 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:17.027559 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk47f\" (UniqueName: \"kubernetes.io/projected/ba7a6633-133a-4815-8cda-5653df7c9535-kube-api-access-kk47f\") pod \"payload-processing-5bf47cd988-774v7\" (UID: \"ba7a6633-133a-4815-8cda-5653df7c9535\") " pod="openshift-ingress/payload-processing-5bf47cd988-774v7" Jul 21 02:18:17.027715 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:17.027631 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed4564f1-d28f-471e-9f52-c2fb73e5543d-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-222pw\" (UID: \"ed4564f1-d28f-471e-9f52-c2fb73e5543d\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-222pw" Jul 21 02:18:17.027794 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:17.027725 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba7a6633-133a-4815-8cda-5653df7c9535-plugins-config-volume\") pod \"payload-processing-5bf47cd988-774v7\" (UID: \"ba7a6633-133a-4815-8cda-5653df7c9535\") " pod="openshift-ingress/payload-processing-5bf47cd988-774v7" Jul 21 02:18:17.027875 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:17.027845 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc45l\" (UniqueName: \"kubernetes.io/projected/ed4564f1-d28f-471e-9f52-c2fb73e5543d-kube-api-access-sc45l\") pod \"payload-pre-processing-7677b774bb-222pw\" (UID: \"ed4564f1-d28f-471e-9f52-c2fb73e5543d\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-222pw" Jul 21 02:18:17.097778 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:17.097745 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55f777c84-9xkgr" event={"ID":"32e2eb32-3083-4856-a8b5-97eeaa74175d","Type":"ContainerStarted","Data":"a45b912728b863652fac152d1eb2f0d87f8ee5be07a1151ee6e877f1fee4a48e"} Jul 21 02:18:17.129234 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:17.129165 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba7a6633-133a-4815-8cda-5653df7c9535-plugins-config-volume\") pod \"payload-processing-5bf47cd988-774v7\" (UID: \"ba7a6633-133a-4815-8cda-5653df7c9535\") " pod="openshift-ingress/payload-processing-5bf47cd988-774v7" Jul 21 02:18:17.129388 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:17.129238 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc45l\" (UniqueName: \"kubernetes.io/projected/ed4564f1-d28f-471e-9f52-c2fb73e5543d-kube-api-access-sc45l\") pod \"payload-pre-processing-7677b774bb-222pw\" (UID: \"ed4564f1-d28f-471e-9f52-c2fb73e5543d\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-222pw" Jul 21 02:18:17.129388 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:17.129291 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk47f\" (UniqueName: \"kubernetes.io/projected/ba7a6633-133a-4815-8cda-5653df7c9535-kube-api-access-kk47f\") pod \"payload-processing-5bf47cd988-774v7\" (UID: \"ba7a6633-133a-4815-8cda-5653df7c9535\") " pod="openshift-ingress/payload-processing-5bf47cd988-774v7" Jul 21 02:18:17.129388 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:17.129337 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed4564f1-d28f-471e-9f52-c2fb73e5543d-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-222pw\" (UID: \"ed4564f1-d28f-471e-9f52-c2fb73e5543d\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-222pw" Jul 21 02:18:17.130079 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:17.130055 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba7a6633-133a-4815-8cda-5653df7c9535-plugins-config-volume\") pod \"payload-processing-5bf47cd988-774v7\" (UID: \"ba7a6633-133a-4815-8cda-5653df7c9535\") " pod="openshift-ingress/payload-processing-5bf47cd988-774v7" Jul 21 02:18:17.130117 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:17.130103 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed4564f1-d28f-471e-9f52-c2fb73e5543d-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-222pw\" (UID: \"ed4564f1-d28f-471e-9f52-c2fb73e5543d\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-222pw" Jul 21 02:18:17.148046 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:17.148023 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc45l\" (UniqueName: \"kubernetes.io/projected/ed4564f1-d28f-471e-9f52-c2fb73e5543d-kube-api-access-sc45l\") pod \"payload-pre-processing-7677b774bb-222pw\" (UID: \"ed4564f1-d28f-471e-9f52-c2fb73e5543d\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-222pw" Jul 21 02:18:17.148716 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:17.148695 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk47f\" (UniqueName: \"kubernetes.io/projected/ba7a6633-133a-4815-8cda-5653df7c9535-kube-api-access-kk47f\") pod \"payload-processing-5bf47cd988-774v7\" (UID: \"ba7a6633-133a-4815-8cda-5653df7c9535\") " pod="openshift-ingress/payload-processing-5bf47cd988-774v7" Jul 21 02:18:17.189601 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:17.189575 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-774v7" Jul 21 02:18:17.260118 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:17.260078 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-222pw" Jul 21 02:18:17.437240 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:17.437174 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-222pw"] Jul 21 02:18:17.439950 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:18:17.439918 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded4564f1_d28f_471e_9f52_c2fb73e5543d.slice/crio-d4d2076006fe2a8d08d883ccbfd2b20794c67df24235a280bcc6c233470fcdb0 WatchSource:0}: Error finding container d4d2076006fe2a8d08d883ccbfd2b20794c67df24235a280bcc6c233470fcdb0: Status 404 returned error can't find the container with id d4d2076006fe2a8d08d883ccbfd2b20794c67df24235a280bcc6c233470fcdb0 Jul 21 02:18:17.557224 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:17.557195 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-774v7"] Jul 21 02:18:17.558847 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:18:17.558822 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba7a6633_133a_4815_8cda_5653df7c9535.slice/crio-164f611a05ffd28a4077f571be9c7eb5884200d775550f380b962cd754a7ad5b WatchSource:0}: Error finding container 164f611a05ffd28a4077f571be9c7eb5884200d775550f380b962cd754a7ad5b: Status 404 returned error can't find the container with id 164f611a05ffd28a4077f571be9c7eb5884200d775550f380b962cd754a7ad5b Jul 21 02:18:18.104127 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:18.104087 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-222pw" event={"ID":"ed4564f1-d28f-471e-9f52-c2fb73e5543d","Type":"ContainerStarted","Data":"d4d2076006fe2a8d08d883ccbfd2b20794c67df24235a280bcc6c233470fcdb0"} Jul 21 02:18:18.105785 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:18.105755 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-774v7" event={"ID":"ba7a6633-133a-4815-8cda-5653df7c9535","Type":"ContainerStarted","Data":"164f611a05ffd28a4077f571be9c7eb5884200d775550f380b962cd754a7ad5b"} Jul 21 02:18:19.113795 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:19.113754 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55f777c84-9xkgr" event={"ID":"32e2eb32-3083-4856-a8b5-97eeaa74175d","Type":"ContainerStarted","Data":"3ce1359c93793bceeedeb7ed921c7a41a6fcc2e410dc18185417704bf002b150"} Jul 21 02:18:19.114355 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:19.113934 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-55f777c84-9xkgr" Jul 21 02:18:19.135534 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:19.135481 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-55f777c84-9xkgr" podStartSLOduration=2.365228716 podStartE2EDuration="4.135466468s" totalImagesPullingTime="1.770237752s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 02:18:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:18:19.133188998 +0000 UTC m=+533.152682903" watchObservedRunningTime="2026-07-21 02:18:19.135466468 +0000 UTC m=+533.154960371" Jul 21 02:18:20.120185 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:20.120153 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-222pw" event={"ID":"ed4564f1-d28f-471e-9f52-c2fb73e5543d","Type":"ContainerStarted","Data":"2963fff9133f2d09861d5fca333af50bcbc5debc5d28a6b812680ee4d356d159"} Jul 21 02:18:20.120611 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:20.120311 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-222pw" Jul 21 02:18:20.121664 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:20.121645 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-774v7" event={"ID":"ba7a6633-133a-4815-8cda-5653df7c9535","Type":"ContainerStarted","Data":"6a28be2598537ca179b5816fa03e04e4cc4d7ebb3e0bcdc7a0abcf649e86dbc0"} Jul 21 02:18:20.121744 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:20.121683 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5bf47cd988-774v7" Jul 21 02:18:20.137716 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:20.137675 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7677b774bb-222pw" podStartSLOduration=2.044029879 podStartE2EDuration="4.13766215s" totalImagesPullingTime="2.093632271s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 02:18:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:18:20.136719844 +0000 UTC m=+534.156213761" watchObservedRunningTime="2026-07-21 02:18:20.13766215 +0000 UTC m=+534.157156051" Jul 21 02:18:20.155129 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:20.155087 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5bf47cd988-774v7" podStartSLOduration=2.186387792 podStartE2EDuration="4.155074252s" totalImagesPullingTime="1.96868646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 02:18:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:18:20.15366015 +0000 UTC m=+534.173154077" watchObservedRunningTime="2026-07-21 02:18:20.155074252 +0000 UTC m=+534.174568155" Jul 21 02:18:25.126209 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:25.126181 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-55f777c84-9xkgr" Jul 21 02:18:31.127909 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:31.127878 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5bf47cd988-774v7" Jul 21 02:18:31.128370 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:31.127971 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-222pw" Jul 21 02:18:43.695788 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:43.695754 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9kxvh"] Jul 21 02:18:43.699389 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:43.699369 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9kxvh" Jul 21 02:18:43.702058 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:43.702037 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 21 02:18:43.703112 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:43.703089 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-64ltv\"" Jul 21 02:18:43.703112 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:43.703103 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 21 02:18:43.703311 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:43.703090 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 21 02:18:43.716606 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:43.716562 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9kxvh"] Jul 21 02:18:43.767543 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:43.767514 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a331a8b-a1e1-41ce-89e2-5faab3b0f87d-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9kxvh\" (UID: \"9a331a8b-a1e1-41ce-89e2-5faab3b0f87d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9kxvh" Jul 21 02:18:43.767702 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:43.767555 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a331a8b-a1e1-41ce-89e2-5faab3b0f87d-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9kxvh\" (UID: \"9a331a8b-a1e1-41ce-89e2-5faab3b0f87d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9kxvh" Jul 21 02:18:43.767702 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:43.767578 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a331a8b-a1e1-41ce-89e2-5faab3b0f87d-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9kxvh\" (UID: \"9a331a8b-a1e1-41ce-89e2-5faab3b0f87d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9kxvh" Jul 21 02:18:43.767702 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:43.767656 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxmn4\" (UniqueName: \"kubernetes.io/projected/9a331a8b-a1e1-41ce-89e2-5faab3b0f87d-kube-api-access-nxmn4\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9kxvh\" (UID: \"9a331a8b-a1e1-41ce-89e2-5faab3b0f87d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9kxvh" Jul 21 02:18:43.767702 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:43.767688 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a331a8b-a1e1-41ce-89e2-5faab3b0f87d-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9kxvh\" (UID: \"9a331a8b-a1e1-41ce-89e2-5faab3b0f87d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9kxvh" Jul 21 02:18:43.767830 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:43.767724 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a331a8b-a1e1-41ce-89e2-5faab3b0f87d-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9kxvh\" (UID: \"9a331a8b-a1e1-41ce-89e2-5faab3b0f87d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9kxvh" Jul 21 02:18:43.869154 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:43.869115 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a331a8b-a1e1-41ce-89e2-5faab3b0f87d-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9kxvh\" (UID: \"9a331a8b-a1e1-41ce-89e2-5faab3b0f87d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9kxvh" Jul 21 02:18:43.869358 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:43.869187 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a331a8b-a1e1-41ce-89e2-5faab3b0f87d-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9kxvh\" (UID: \"9a331a8b-a1e1-41ce-89e2-5faab3b0f87d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9kxvh" Jul 21 02:18:43.869358 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:43.869226 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a331a8b-a1e1-41ce-89e2-5faab3b0f87d-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9kxvh\" (UID: \"9a331a8b-a1e1-41ce-89e2-5faab3b0f87d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9kxvh" Jul 21 02:18:43.869358 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:43.869252 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a331a8b-a1e1-41ce-89e2-5faab3b0f87d-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9kxvh\" (UID: \"9a331a8b-a1e1-41ce-89e2-5faab3b0f87d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9kxvh" Jul 21 02:18:43.869358 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:43.869327 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxmn4\" (UniqueName: \"kubernetes.io/projected/9a331a8b-a1e1-41ce-89e2-5faab3b0f87d-kube-api-access-nxmn4\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9kxvh\" (UID: \"9a331a8b-a1e1-41ce-89e2-5faab3b0f87d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9kxvh" Jul 21 02:18:43.869358 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:43.869357 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a331a8b-a1e1-41ce-89e2-5faab3b0f87d-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9kxvh\" (UID: \"9a331a8b-a1e1-41ce-89e2-5faab3b0f87d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9kxvh" Jul 21 02:18:43.869628 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:43.869572 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a331a8b-a1e1-41ce-89e2-5faab3b0f87d-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9kxvh\" (UID: \"9a331a8b-a1e1-41ce-89e2-5faab3b0f87d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9kxvh" Jul 21 02:18:43.869683 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:43.869639 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a331a8b-a1e1-41ce-89e2-5faab3b0f87d-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9kxvh\" (UID: \"9a331a8b-a1e1-41ce-89e2-5faab3b0f87d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9kxvh" Jul 21 02:18:43.869737 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:43.869685 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a331a8b-a1e1-41ce-89e2-5faab3b0f87d-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9kxvh\" (UID: \"9a331a8b-a1e1-41ce-89e2-5faab3b0f87d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9kxvh" Jul 21 02:18:43.871434 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:43.871407 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a331a8b-a1e1-41ce-89e2-5faab3b0f87d-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9kxvh\" (UID: \"9a331a8b-a1e1-41ce-89e2-5faab3b0f87d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9kxvh" Jul 21 02:18:43.871682 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:43.871662 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a331a8b-a1e1-41ce-89e2-5faab3b0f87d-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9kxvh\" (UID: \"9a331a8b-a1e1-41ce-89e2-5faab3b0f87d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9kxvh" Jul 21 02:18:43.880002 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:43.879977 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxmn4\" (UniqueName: \"kubernetes.io/projected/9a331a8b-a1e1-41ce-89e2-5faab3b0f87d-kube-api-access-nxmn4\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9kxvh\" (UID: \"9a331a8b-a1e1-41ce-89e2-5faab3b0f87d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9kxvh" Jul 21 02:18:44.010622 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:44.010529 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9kxvh" Jul 21 02:18:44.146228 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:44.146200 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9kxvh"] Jul 21 02:18:44.148599 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:18:44.148571 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a331a8b_a1e1_41ce_89e2_5faab3b0f87d.slice/crio-47040292e3b41dc42feadd708bd0cd13e65f7f950f25c64c989a47fb56e02a5b WatchSource:0}: Error finding container 47040292e3b41dc42feadd708bd0cd13e65f7f950f25c64c989a47fb56e02a5b: Status 404 returned error can't find the container with id 47040292e3b41dc42feadd708bd0cd13e65f7f950f25c64c989a47fb56e02a5b Jul 21 02:18:44.237478 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:44.237437 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9kxvh" event={"ID":"9a331a8b-a1e1-41ce-89e2-5faab3b0f87d","Type":"ContainerStarted","Data":"47040292e3b41dc42feadd708bd0cd13e65f7f950f25c64c989a47fb56e02a5b"} Jul 21 02:18:46.249157 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:46.249118 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9kxvh" event={"ID":"9a331a8b-a1e1-41ce-89e2-5faab3b0f87d","Type":"ContainerStarted","Data":"061af3150d5d54cae93fd5c44b568177840ccb7b034db74ee2054f87cc641b15"} Jul 21 02:18:46.267534 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:46.267478 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9kxvh" podStartSLOduration=2.143180625 podStartE2EDuration="3.267459096s" totalImagesPullingTime="1.124278471s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 02:18:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:18:46.266314743 +0000 UTC m=+560.285808648" watchObservedRunningTime="2026-07-21 02:18:46.267459096 +0000 UTC m=+560.286953002" Jul 21 02:18:54.011820 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:54.011780 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9kxvh" Jul 21 02:18:54.012345 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:54.011850 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9kxvh" Jul 21 02:18:54.024385 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:54.024356 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9kxvh" Jul 21 02:18:54.298962 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:18:54.298878 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9kxvh" Jul 21 02:19:04.228491 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:19:04.228449 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-dcxnd"] Jul 21 02:19:04.232215 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:19:04.232193 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dcxnd" Jul 21 02:19:04.234961 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:19:04.234931 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 21 02:19:04.252704 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:19:04.252668 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-dcxnd"] Jul 21 02:19:04.372500 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:19:04.372469 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3100ae84-5919-4101-ba04-f4ce37ce4c14-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dcxnd\" (UID: \"3100ae84-5919-4101-ba04-f4ce37ce4c14\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dcxnd" Jul 21 02:19:04.372712 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:19:04.372506 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3100ae84-5919-4101-ba04-f4ce37ce4c14-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dcxnd\" (UID: \"3100ae84-5919-4101-ba04-f4ce37ce4c14\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dcxnd" Jul 21 02:19:04.372712 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:19:04.372532 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3100ae84-5919-4101-ba04-f4ce37ce4c14-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dcxnd\" (UID: \"3100ae84-5919-4101-ba04-f4ce37ce4c14\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dcxnd" Jul 21 02:19:04.372712 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:19:04.372556 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd25t\" (UniqueName: \"kubernetes.io/projected/3100ae84-5919-4101-ba04-f4ce37ce4c14-kube-api-access-sd25t\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dcxnd\" (UID: \"3100ae84-5919-4101-ba04-f4ce37ce4c14\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dcxnd" Jul 21 02:19:04.372712 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:19:04.372608 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3100ae84-5919-4101-ba04-f4ce37ce4c14-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dcxnd\" (UID: \"3100ae84-5919-4101-ba04-f4ce37ce4c14\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dcxnd" Jul 21 02:19:04.372712 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:19:04.372626 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3100ae84-5919-4101-ba04-f4ce37ce4c14-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dcxnd\" (UID: \"3100ae84-5919-4101-ba04-f4ce37ce4c14\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dcxnd" Jul 21 02:19:04.474041 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:19:04.473998 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3100ae84-5919-4101-ba04-f4ce37ce4c14-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dcxnd\" (UID: \"3100ae84-5919-4101-ba04-f4ce37ce4c14\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dcxnd" Jul 21 02:19:04.474041 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:19:04.474042 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3100ae84-5919-4101-ba04-f4ce37ce4c14-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dcxnd\" (UID: \"3100ae84-5919-4101-ba04-f4ce37ce4c14\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dcxnd" Jul 21 02:19:04.474041 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:19:04.474063 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3100ae84-5919-4101-ba04-f4ce37ce4c14-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dcxnd\" (UID: \"3100ae84-5919-4101-ba04-f4ce37ce4c14\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dcxnd" Jul 21 02:19:04.474041 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:19:04.474084 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd25t\" (UniqueName: \"kubernetes.io/projected/3100ae84-5919-4101-ba04-f4ce37ce4c14-kube-api-access-sd25t\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dcxnd\" (UID: \"3100ae84-5919-4101-ba04-f4ce37ce4c14\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dcxnd" Jul 21 02:19:04.474464 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:19:04.474112 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3100ae84-5919-4101-ba04-f4ce37ce4c14-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dcxnd\" (UID: \"3100ae84-5919-4101-ba04-f4ce37ce4c14\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dcxnd" Jul 21 02:19:04.474464 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:19:04.474127 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3100ae84-5919-4101-ba04-f4ce37ce4c14-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dcxnd\" (UID: \"3100ae84-5919-4101-ba04-f4ce37ce4c14\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dcxnd" Jul 21 02:19:04.474565 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:19:04.474544 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3100ae84-5919-4101-ba04-f4ce37ce4c14-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dcxnd\" (UID: \"3100ae84-5919-4101-ba04-f4ce37ce4c14\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dcxnd" Jul 21 02:19:04.474608 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:19:04.474592 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3100ae84-5919-4101-ba04-f4ce37ce4c14-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dcxnd\" (UID: \"3100ae84-5919-4101-ba04-f4ce37ce4c14\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dcxnd" Jul 21 02:19:04.474642 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:19:04.474592 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3100ae84-5919-4101-ba04-f4ce37ce4c14-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dcxnd\" (UID: \"3100ae84-5919-4101-ba04-f4ce37ce4c14\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dcxnd" Jul 21 02:19:04.476479 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:19:04.476458 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3100ae84-5919-4101-ba04-f4ce37ce4c14-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dcxnd\" (UID: \"3100ae84-5919-4101-ba04-f4ce37ce4c14\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dcxnd" Jul 21 02:19:04.476753 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:19:04.476734 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3100ae84-5919-4101-ba04-f4ce37ce4c14-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dcxnd\" (UID: \"3100ae84-5919-4101-ba04-f4ce37ce4c14\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dcxnd" Jul 21 02:19:04.487438 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:19:04.487372 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd25t\" (UniqueName: \"kubernetes.io/projected/3100ae84-5919-4101-ba04-f4ce37ce4c14-kube-api-access-sd25t\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dcxnd\" (UID: \"3100ae84-5919-4101-ba04-f4ce37ce4c14\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dcxnd" Jul 21 02:19:04.542823 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:19:04.542775 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dcxnd" Jul 21 02:19:04.686059 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:19:04.686030 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-dcxnd"] Jul 21 02:19:04.688764 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:19:04.688737 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3100ae84_5919_4101_ba04_f4ce37ce4c14.slice/crio-e9df0e317172f2fa9dbcbb2fe1d2daf0926f118a833dd7bb4a9f142b07b1eeb3 WatchSource:0}: Error finding container e9df0e317172f2fa9dbcbb2fe1d2daf0926f118a833dd7bb4a9f142b07b1eeb3: Status 404 returned error can't find the container with id e9df0e317172f2fa9dbcbb2fe1d2daf0926f118a833dd7bb4a9f142b07b1eeb3 Jul 21 02:19:05.345051 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:19:05.345013 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dcxnd" event={"ID":"3100ae84-5919-4101-ba04-f4ce37ce4c14","Type":"ContainerStarted","Data":"68a6d665c6d170cae4206daef3a5d25ce05d4d935fd1b4487ce948f8b98acccb"} Jul 21 02:19:05.345478 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:19:05.345073 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dcxnd" event={"ID":"3100ae84-5919-4101-ba04-f4ce37ce4c14","Type":"ContainerStarted","Data":"e9df0e317172f2fa9dbcbb2fe1d2daf0926f118a833dd7bb4a9f142b07b1eeb3"} Jul 21 02:19:05.367321 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:19:05.367251 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dcxnd" podStartSLOduration=1.186564131 podStartE2EDuration="1.367234315s" totalImagesPullingTime="180.670184ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 02:19:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:19:05.365568974 +0000 UTC m=+579.385062880" watchObservedRunningTime="2026-07-21 02:19:05.367234315 +0000 UTC m=+579.386728219" Jul 21 02:19:14.543143 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:19:14.543099 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dcxnd" Jul 21 02:19:14.543523 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:19:14.543183 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dcxnd" Jul 21 02:19:14.555675 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:19:14.555653 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dcxnd" Jul 21 02:19:15.406551 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:19:15.406521 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dcxnd" Jul 21 02:19:15.586222 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:19:15.586190 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-mlpb8"] Jul 21 02:19:15.589416 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:19:15.589397 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mlpb8" Jul 21 02:19:15.592036 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:19:15.592010 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 21 02:19:15.600184 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:19:15.600158 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-mlpb8"] Jul 21 02:19:15.669147 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:19:15.669061 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c38b2c7-6c2e-4f80-8789-3797e210befe-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mlpb8\" (UID: \"0c38b2c7-6c2e-4f80-8789-3797e210befe\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mlpb8" Jul 21 02:19:15.669147 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:19:15.669101 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c38b2c7-6c2e-4f80-8789-3797e210befe-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mlpb8\" (UID: \"0c38b2c7-6c2e-4f80-8789-3797e210befe\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mlpb8" Jul 21 02:19:15.669147 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:19:15.669117 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c38b2c7-6c2e-4f80-8789-3797e210befe-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mlpb8\" (UID: \"0c38b2c7-6c2e-4f80-8789-3797e210befe\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mlpb8" Jul 21 02:19:15.669147 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:19:15.669134 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c38b2c7-6c2e-4f80-8789-3797e210befe-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mlpb8\" (UID: \"0c38b2c7-6c2e-4f80-8789-3797e210befe\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mlpb8" Jul 21 02:19:15.669428 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:19:15.669192 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsr7g\" (UniqueName: \"kubernetes.io/projected/0c38b2c7-6c2e-4f80-8789-3797e210befe-kube-api-access-tsr7g\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mlpb8\" (UID: \"0c38b2c7-6c2e-4f80-8789-3797e210befe\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mlpb8" Jul 21 02:19:15.669428 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:19:15.669236 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c38b2c7-6c2e-4f80-8789-3797e210befe-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mlpb8\" (UID: \"0c38b2c7-6c2e-4f80-8789-3797e210befe\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mlpb8" Jul 21 02:19:15.770656 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:19:15.770617 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c38b2c7-6c2e-4f80-8789-3797e210befe-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mlpb8\" (UID: \"0c38b2c7-6c2e-4f80-8789-3797e210befe\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mlpb8" Jul 21 02:19:15.770656 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:19:15.770655 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c38b2c7-6c2e-4f80-8789-3797e210befe-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mlpb8\" (UID: \"0c38b2c7-6c2e-4f80-8789-3797e210befe\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mlpb8" Jul 21 02:19:15.770899 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:19:15.770674 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c38b2c7-6c2e-4f80-8789-3797e210befe-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mlpb8\" (UID: \"0c38b2c7-6c2e-4f80-8789-3797e210befe\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mlpb8" Jul 21 02:19:15.770899 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:19:15.770689 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c38b2c7-6c2e-4f80-8789-3797e210befe-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mlpb8\" (UID: \"0c38b2c7-6c2e-4f80-8789-3797e210befe\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mlpb8" Jul 21 02:19:15.770899 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:19:15.770726 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsr7g\" (UniqueName: \"kubernetes.io/projected/0c38b2c7-6c2e-4f80-8789-3797e210befe-kube-api-access-tsr7g\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mlpb8\" (UID: \"0c38b2c7-6c2e-4f80-8789-3797e210befe\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mlpb8" Jul 21 02:19:15.770899 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:19:15.770769 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c38b2c7-6c2e-4f80-8789-3797e210befe-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mlpb8\" (UID: \"0c38b2c7-6c2e-4f80-8789-3797e210befe\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mlpb8" Jul 21 02:19:15.771103 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:19:15.771089 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c38b2c7-6c2e-4f80-8789-3797e210befe-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mlpb8\" (UID: \"0c38b2c7-6c2e-4f80-8789-3797e210befe\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mlpb8" Jul 21 02:19:15.771159 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:19:15.771122 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c38b2c7-6c2e-4f80-8789-3797e210befe-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mlpb8\" (UID: \"0c38b2c7-6c2e-4f80-8789-3797e210befe\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mlpb8" Jul 21 02:19:15.771211 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:19:15.771153 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c38b2c7-6c2e-4f80-8789-3797e210befe-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mlpb8\" (UID: \"0c38b2c7-6c2e-4f80-8789-3797e210befe\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mlpb8" Jul 21 02:19:15.773077 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:19:15.773050 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c38b2c7-6c2e-4f80-8789-3797e210befe-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mlpb8\" (UID: \"0c38b2c7-6c2e-4f80-8789-3797e210befe\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mlpb8" Jul 21 02:19:15.773294 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:19:15.773259 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c38b2c7-6c2e-4f80-8789-3797e210befe-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mlpb8\" (UID: \"0c38b2c7-6c2e-4f80-8789-3797e210befe\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mlpb8" Jul 21 02:19:15.779114 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:19:15.779097 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsr7g\" (UniqueName: \"kubernetes.io/projected/0c38b2c7-6c2e-4f80-8789-3797e210befe-kube-api-access-tsr7g\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mlpb8\" (UID: \"0c38b2c7-6c2e-4f80-8789-3797e210befe\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mlpb8" Jul 21 02:19:15.901726 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:19:15.901692 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mlpb8" Jul 21 02:19:16.042347 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:19:16.042241 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-mlpb8"] Jul 21 02:19:16.045294 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:19:16.045245 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c38b2c7_6c2e_4f80_8789_3797e210befe.slice/crio-d103b973c80353b1699126532df9dab6336149666867c11d46a3caa90d333b5c WatchSource:0}: Error finding container d103b973c80353b1699126532df9dab6336149666867c11d46a3caa90d333b5c: Status 404 returned error can't find the container with id d103b973c80353b1699126532df9dab6336149666867c11d46a3caa90d333b5c Jul 21 02:19:16.401972 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:19:16.401936 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mlpb8" event={"ID":"0c38b2c7-6c2e-4f80-8789-3797e210befe","Type":"ContainerStarted","Data":"a6146f590376940cd980492b40af7f28aea20fd1fe351488175ae802934b1fda"} Jul 21 02:19:16.402139 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:19:16.401998 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mlpb8" event={"ID":"0c38b2c7-6c2e-4f80-8789-3797e210befe","Type":"ContainerStarted","Data":"d103b973c80353b1699126532df9dab6336149666867c11d46a3caa90d333b5c"} Jul 21 02:19:16.420716 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:19:16.420667 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mlpb8" podStartSLOduration=1.2630038510000001 podStartE2EDuration="1.420653122s" totalImagesPullingTime="157.649271ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 02:19:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:19:16.419647929 +0000 UTC m=+590.439141853" watchObservedRunningTime="2026-07-21 02:19:16.420653122 +0000 UTC m=+590.440147026" Jul 21 02:19:25.901813 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:19:25.901776 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mlpb8" Jul 21 02:19:25.902419 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:19:25.901852 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mlpb8" Jul 21 02:19:25.914162 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:19:25.914140 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mlpb8" Jul 21 02:19:26.464006 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:19:26.463974 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mlpb8" Jul 21 02:19:28.346900 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:19:28.346873 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qv9kr_bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9/ovn-acl-logging/0.log" Jul 21 02:19:28.347381 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:19:28.347367 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qv9kr_bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9/ovn-acl-logging/0.log" Jul 21 02:21:28.217219 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:21:28.217143 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-65f68477d7-rtzbp"] Jul 21 02:21:28.217650 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:21:28.217360 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-65f68477d7-rtzbp" podUID="9511c95c-704b-49d4-abb6-5db8cb838e08" containerName="manager" containerID="cri-o://7f1143780fb4614af7296dbe6e205b3f0395d0564289b7d5589fa2f60423d47c" gracePeriod=30 Jul 21 02:21:28.475861 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:21:28.475805 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65f68477d7-rtzbp" Jul 21 02:21:28.641395 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:21:28.641363 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwk4c\" (UniqueName: \"kubernetes.io/projected/9511c95c-704b-49d4-abb6-5db8cb838e08-kube-api-access-fwk4c\") pod \"9511c95c-704b-49d4-abb6-5db8cb838e08\" (UID: \"9511c95c-704b-49d4-abb6-5db8cb838e08\") " Jul 21 02:21:28.641572 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:21:28.641450 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9511c95c-704b-49d4-abb6-5db8cb838e08-webhook-cert\") pod \"9511c95c-704b-49d4-abb6-5db8cb838e08\" (UID: \"9511c95c-704b-49d4-abb6-5db8cb838e08\") " Jul 21 02:21:28.643382 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:21:28.643343 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9511c95c-704b-49d4-abb6-5db8cb838e08-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "9511c95c-704b-49d4-abb6-5db8cb838e08" (UID: "9511c95c-704b-49d4-abb6-5db8cb838e08"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 02:21:28.643382 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:21:28.643356 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9511c95c-704b-49d4-abb6-5db8cb838e08-kube-api-access-fwk4c" (OuterVolumeSpecName: "kube-api-access-fwk4c") pod "9511c95c-704b-49d4-abb6-5db8cb838e08" (UID: "9511c95c-704b-49d4-abb6-5db8cb838e08"). InnerVolumeSpecName "kube-api-access-fwk4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 02:21:28.743129 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:21:28.743068 2574 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9511c95c-704b-49d4-abb6-5db8cb838e08-webhook-cert\") on node \"ip-10-0-141-37.ec2.internal\" DevicePath \"\"" Jul 21 02:21:28.743129 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:21:28.743094 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwk4c\" (UniqueName: \"kubernetes.io/projected/9511c95c-704b-49d4-abb6-5db8cb838e08-kube-api-access-fwk4c\") on node \"ip-10-0-141-37.ec2.internal\" DevicePath \"\"" Jul 21 02:21:29.044369 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:21:29.044259 2574 generic.go:356] "Generic (PLEG): container finished" podID="9511c95c-704b-49d4-abb6-5db8cb838e08" containerID="7f1143780fb4614af7296dbe6e205b3f0395d0564289b7d5589fa2f60423d47c" exitCode=0 Jul 21 02:21:29.044369 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:21:29.044321 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65f68477d7-rtzbp" Jul 21 02:21:29.044369 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:21:29.044328 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65f68477d7-rtzbp" event={"ID":"9511c95c-704b-49d4-abb6-5db8cb838e08","Type":"ContainerDied","Data":"7f1143780fb4614af7296dbe6e205b3f0395d0564289b7d5589fa2f60423d47c"} Jul 21 02:21:29.044369 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:21:29.044370 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65f68477d7-rtzbp" event={"ID":"9511c95c-704b-49d4-abb6-5db8cb838e08","Type":"ContainerDied","Data":"0e4344ebe1d1893ebf8f069721a2e7331ab8f3c6c51ff7d3c954318b8e140d11"} Jul 21 02:21:29.044676 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:21:29.044390 2574 scope.go:117] "RemoveContainer" containerID="7f1143780fb4614af7296dbe6e205b3f0395d0564289b7d5589fa2f60423d47c" Jul 21 02:21:29.053158 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:21:29.053141 2574 scope.go:117] "RemoveContainer" containerID="7f1143780fb4614af7296dbe6e205b3f0395d0564289b7d5589fa2f60423d47c" Jul 21 02:21:29.053431 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:21:29.053411 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f1143780fb4614af7296dbe6e205b3f0395d0564289b7d5589fa2f60423d47c\": container with ID starting with 7f1143780fb4614af7296dbe6e205b3f0395d0564289b7d5589fa2f60423d47c not found: ID does not exist" containerID="7f1143780fb4614af7296dbe6e205b3f0395d0564289b7d5589fa2f60423d47c" Jul 21 02:21:29.053498 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:21:29.053439 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f1143780fb4614af7296dbe6e205b3f0395d0564289b7d5589fa2f60423d47c"} err="failed to get container status \"7f1143780fb4614af7296dbe6e205b3f0395d0564289b7d5589fa2f60423d47c\": rpc error: code = NotFound desc = could not find container \"7f1143780fb4614af7296dbe6e205b3f0395d0564289b7d5589fa2f60423d47c\": container with ID starting with 7f1143780fb4614af7296dbe6e205b3f0395d0564289b7d5589fa2f60423d47c not found: ID does not exist" Jul 21 02:21:29.086404 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:21:29.086386 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-65f68477d7-rtzbp"] Jul 21 02:21:29.093868 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:21:29.093845 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-65f68477d7-rtzbp"] Jul 21 02:21:30.484353 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:21:30.484325 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9511c95c-704b-49d4-abb6-5db8cb838e08" path="/var/lib/kubelet/pods/9511c95c-704b-49d4-abb6-5db8cb838e08/volumes" Jul 21 02:24:28.375874 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:24:28.375807 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qv9kr_bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9/ovn-acl-logging/0.log" Jul 21 02:24:28.377041 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:24:28.377020 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qv9kr_bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9/ovn-acl-logging/0.log" Jul 21 02:27:40.551496 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:27:40.551421 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hp5wh"] Jul 21 02:27:40.553975 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:27:40.551620 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hp5wh" podUID="59a1b2d9-71d2-43fd-bbc1-a8936442e3d6" containerName="manager" containerID="cri-o://3c2145dd5c8682245f19025b11a5303ea992b03426e262d65da2d300f0964b7d" gracePeriod=10 Jul 21 02:27:43.206732 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:27:43.206704 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hp5wh" Jul 21 02:27:43.256147 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:27:43.256108 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/59a1b2d9-71d2-43fd-bbc1-a8936442e3d6-extensions-socket-volume\") pod \"59a1b2d9-71d2-43fd-bbc1-a8936442e3d6\" (UID: \"59a1b2d9-71d2-43fd-bbc1-a8936442e3d6\") " Jul 21 02:27:43.256348 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:27:43.256188 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sl7nk\" (UniqueName: \"kubernetes.io/projected/59a1b2d9-71d2-43fd-bbc1-a8936442e3d6-kube-api-access-sl7nk\") pod \"59a1b2d9-71d2-43fd-bbc1-a8936442e3d6\" (UID: \"59a1b2d9-71d2-43fd-bbc1-a8936442e3d6\") " Jul 21 02:27:43.256562 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:27:43.256539 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59a1b2d9-71d2-43fd-bbc1-a8936442e3d6-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "59a1b2d9-71d2-43fd-bbc1-a8936442e3d6" (UID: "59a1b2d9-71d2-43fd-bbc1-a8936442e3d6"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 02:27:43.258179 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:27:43.258148 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59a1b2d9-71d2-43fd-bbc1-a8936442e3d6-kube-api-access-sl7nk" (OuterVolumeSpecName: "kube-api-access-sl7nk") pod "59a1b2d9-71d2-43fd-bbc1-a8936442e3d6" (UID: "59a1b2d9-71d2-43fd-bbc1-a8936442e3d6"). InnerVolumeSpecName "kube-api-access-sl7nk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 02:27:43.357001 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:27:43.356912 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sl7nk\" (UniqueName: \"kubernetes.io/projected/59a1b2d9-71d2-43fd-bbc1-a8936442e3d6-kube-api-access-sl7nk\") on node \"ip-10-0-141-37.ec2.internal\" DevicePath \"\"" Jul 21 02:27:43.357001 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:27:43.356945 2574 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/59a1b2d9-71d2-43fd-bbc1-a8936442e3d6-extensions-socket-volume\") on node \"ip-10-0-141-37.ec2.internal\" DevicePath \"\"" Jul 21 02:27:43.915417 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:27:43.915385 2574 generic.go:356] "Generic (PLEG): container finished" podID="59a1b2d9-71d2-43fd-bbc1-a8936442e3d6" containerID="3c2145dd5c8682245f19025b11a5303ea992b03426e262d65da2d300f0964b7d" exitCode=0 Jul 21 02:27:43.915589 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:27:43.915423 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hp5wh" Jul 21 02:27:43.915589 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:27:43.915450 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hp5wh" event={"ID":"59a1b2d9-71d2-43fd-bbc1-a8936442e3d6","Type":"ContainerDied","Data":"3c2145dd5c8682245f19025b11a5303ea992b03426e262d65da2d300f0964b7d"} Jul 21 02:27:43.915589 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:27:43.915495 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hp5wh" event={"ID":"59a1b2d9-71d2-43fd-bbc1-a8936442e3d6","Type":"ContainerDied","Data":"0bb18b7e77c4434da9ebd9e4d6b2fa7884219a345c18ace17ec1fe3c17ad1f6e"} Jul 21 02:27:43.915589 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:27:43.915520 2574 scope.go:117] "RemoveContainer" containerID="3c2145dd5c8682245f19025b11a5303ea992b03426e262d65da2d300f0964b7d" Jul 21 02:27:43.925432 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:27:43.925409 2574 scope.go:117] "RemoveContainer" containerID="3c2145dd5c8682245f19025b11a5303ea992b03426e262d65da2d300f0964b7d" Jul 21 02:27:43.925707 ip-10-0-141-37 kubenswrapper[2574]: E0721 02:27:43.925687 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c2145dd5c8682245f19025b11a5303ea992b03426e262d65da2d300f0964b7d\": container with ID starting with 3c2145dd5c8682245f19025b11a5303ea992b03426e262d65da2d300f0964b7d not found: ID does not exist" containerID="3c2145dd5c8682245f19025b11a5303ea992b03426e262d65da2d300f0964b7d" Jul 21 02:27:43.925769 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:27:43.925719 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c2145dd5c8682245f19025b11a5303ea992b03426e262d65da2d300f0964b7d"} err="failed to get container status \"3c2145dd5c8682245f19025b11a5303ea992b03426e262d65da2d300f0964b7d\": rpc error: code = NotFound desc = could not find container \"3c2145dd5c8682245f19025b11a5303ea992b03426e262d65da2d300f0964b7d\": container with ID starting with 3c2145dd5c8682245f19025b11a5303ea992b03426e262d65da2d300f0964b7d not found: ID does not exist" Jul 21 02:27:43.941926 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:27:43.941895 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hp5wh"] Jul 21 02:27:43.947157 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:27:43.947135 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hp5wh"] Jul 21 02:27:44.485151 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:27:44.485109 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59a1b2d9-71d2-43fd-bbc1-a8936442e3d6" path="/var/lib/kubelet/pods/59a1b2d9-71d2-43fd-bbc1-a8936442e3d6/volumes" Jul 21 02:28:41.696860 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:28:41.696824 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-knw9j"] Jul 21 02:28:41.697354 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:28:41.697276 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9511c95c-704b-49d4-abb6-5db8cb838e08" containerName="manager" Jul 21 02:28:41.697354 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:28:41.697289 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="9511c95c-704b-49d4-abb6-5db8cb838e08" containerName="manager" Jul 21 02:28:41.697354 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:28:41.697313 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59a1b2d9-71d2-43fd-bbc1-a8936442e3d6" containerName="manager" Jul 21 02:28:41.697354 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:28:41.697318 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="59a1b2d9-71d2-43fd-bbc1-a8936442e3d6" containerName="manager" Jul 21 02:28:41.697511 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:28:41.697370 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="9511c95c-704b-49d4-abb6-5db8cb838e08" containerName="manager" Jul 21 02:28:41.697511 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:28:41.697378 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="59a1b2d9-71d2-43fd-bbc1-a8936442e3d6" containerName="manager" Jul 21 02:28:41.702538 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:28:41.702522 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-knw9j" Jul 21 02:28:41.704990 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:28:41.704969 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-h8n22\"" Jul 21 02:28:41.713947 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:28:41.713927 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-knw9j"] Jul 21 02:28:41.768524 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:28:41.768500 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stfnm\" (UniqueName: \"kubernetes.io/projected/c5e9de9e-2824-4d34-a651-2ad2c3d5d9ba-kube-api-access-stfnm\") pod \"kuadrant-operator-controller-manager-86878c769f-knw9j\" (UID: \"c5e9de9e-2824-4d34-a651-2ad2c3d5d9ba\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-knw9j" Jul 21 02:28:41.768637 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:28:41.768559 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c5e9de9e-2824-4d34-a651-2ad2c3d5d9ba-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-knw9j\" (UID: \"c5e9de9e-2824-4d34-a651-2ad2c3d5d9ba\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-knw9j" Jul 21 02:28:41.869806 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:28:41.869781 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c5e9de9e-2824-4d34-a651-2ad2c3d5d9ba-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-knw9j\" (UID: \"c5e9de9e-2824-4d34-a651-2ad2c3d5d9ba\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-knw9j" Jul 21 02:28:41.869931 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:28:41.869873 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stfnm\" (UniqueName: \"kubernetes.io/projected/c5e9de9e-2824-4d34-a651-2ad2c3d5d9ba-kube-api-access-stfnm\") pod \"kuadrant-operator-controller-manager-86878c769f-knw9j\" (UID: \"c5e9de9e-2824-4d34-a651-2ad2c3d5d9ba\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-knw9j" Jul 21 02:28:41.870191 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:28:41.870172 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c5e9de9e-2824-4d34-a651-2ad2c3d5d9ba-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-knw9j\" (UID: \"c5e9de9e-2824-4d34-a651-2ad2c3d5d9ba\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-knw9j" Jul 21 02:28:41.880354 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:28:41.880329 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stfnm\" (UniqueName: \"kubernetes.io/projected/c5e9de9e-2824-4d34-a651-2ad2c3d5d9ba-kube-api-access-stfnm\") pod \"kuadrant-operator-controller-manager-86878c769f-knw9j\" (UID: \"c5e9de9e-2824-4d34-a651-2ad2c3d5d9ba\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-knw9j" Jul 21 02:28:42.013596 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:28:42.013513 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-knw9j" Jul 21 02:28:42.175495 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:28:42.175472 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-knw9j"] Jul 21 02:28:42.178213 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:28:42.178186 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5e9de9e_2824_4d34_a651_2ad2c3d5d9ba.slice/crio-128ddf5756ba83081a1fad86f74f9b63c946c07a348a62089620d1f59e4c25ad WatchSource:0}: Error finding container 128ddf5756ba83081a1fad86f74f9b63c946c07a348a62089620d1f59e4c25ad: Status 404 returned error can't find the container with id 128ddf5756ba83081a1fad86f74f9b63c946c07a348a62089620d1f59e4c25ad Jul 21 02:28:42.193031 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:28:42.193006 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-knw9j" event={"ID":"c5e9de9e-2824-4d34-a651-2ad2c3d5d9ba","Type":"ContainerStarted","Data":"128ddf5756ba83081a1fad86f74f9b63c946c07a348a62089620d1f59e4c25ad"} Jul 21 02:28:43.199347 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:28:43.199315 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-knw9j" event={"ID":"c5e9de9e-2824-4d34-a651-2ad2c3d5d9ba","Type":"ContainerStarted","Data":"10da921372c3b6f29a948cbb155056de7980ff736cceea69f26b67726b7cbfcc"} Jul 21 02:28:43.199737 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:28:43.199387 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-knw9j" Jul 21 02:28:43.225216 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:28:43.225169 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-knw9j" podStartSLOduration=2.225152656 podStartE2EDuration="2.225152656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 02:28:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:28:43.222060438 +0000 UTC m=+1157.241554352" watchObservedRunningTime="2026-07-21 02:28:43.225152656 +0000 UTC m=+1157.244646559" Jul 21 02:28:54.206471 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:28:54.206440 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-knw9j" Jul 21 02:29:28.413422 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:29:28.413394 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qv9kr_bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9/ovn-acl-logging/0.log" Jul 21 02:29:28.420189 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:29:28.420168 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qv9kr_bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9/ovn-acl-logging/0.log" Jul 21 02:34:28.444937 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:34:28.444907 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qv9kr_bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9/ovn-acl-logging/0.log" Jul 21 02:34:28.451178 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:34:28.451158 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qv9kr_bf3f6ce0-b6b7-4896-b7b5-5c103e8f36b9/ovn-acl-logging/0.log" Jul 21 02:35:47.092565 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:35:47.092529 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-9x9k7_b0e21853-e493-4e53-9b6f-fdc88f1d766f/manager/0.log" Jul 21 02:35:47.216510 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:35:47.216470 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-pwzb2_54ddeee7-7991-44cb-9c3a-dc445f966bb2/manager/0.log" Jul 21 02:35:47.473900 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:35:47.473877 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-mjtvp_7126c66e-8b6b-4a9b-b93f-62f9d393ad72/server/0.log" Jul 21 02:35:47.597090 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:35:47.597057 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-2pth6_21493e6e-e2a6-41cc-8483-a644130919a2/manager/0.log" Jul 21 02:35:47.711554 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:35:47.711528 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8475997445-2dwbj_c1c91659-2ab2-4743-a85f-648bf5b5f5b8/copy-manifests/0.log" Jul 21 02:35:47.718562 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:35:47.718537 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8475997445-2dwbj_c1c91659-2ab2-4743-a85f-648bf5b5f5b8/manager/0.log" Jul 21 02:35:48.300584 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:35:48.300550 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-55f777c84-9xkgr_32e2eb32-3083-4856-a8b5-97eeaa74175d/maas-api/0.log" Jul 21 02:35:48.530351 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:35:48.530311 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-4bprz_db59b237-fdb3-4c81-86d2-1cfd7fa4cf4a/postgres/0.log" Jul 21 02:35:50.062915 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:35:50.062883 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-4757q_28d4228c-c846-4f2d-b2e3-6d1c4b2298bd/manager/0.log" Jul 21 02:35:50.436101 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:35:50.436007 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-knw9j_c5e9de9e-2824-4d34-a651-2ad2c3d5d9ba/manager/0.log" Jul 21 02:35:51.202593 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:35:51.202558 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-2zbzn_9cbeff22-f3f1-4e40-85c8-02c7971c9c47/discovery/0.log" Jul 21 02:35:51.321291 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:35:51.321243 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d4c7dc87d-52tfh_e2d7f3ea-52ce-4b44-9821-6f34b323f0f9/kube-auth-proxy/0.log" Jul 21 02:35:51.701640 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:35:51.701601 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-222pw_ed4564f1-d28f-471e-9f52-c2fb73e5543d/payload-pre-processing/0.log" Jul 21 02:35:51.847359 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:35:51.847331 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-774v7_ba7a6633-133a-4815-8cda-5653df7c9535/payload-processing/0.log" Jul 21 02:35:52.292394 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:35:52.292352 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-9kxvh_9a331a8b-a1e1-41ce-89e2-5faab3b0f87d/main/0.log" Jul 21 02:35:52.515450 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:35:52.515420 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-dcxnd_3100ae84-5919-4101-ba04-f4ce37ce4c14/main/0.log" Jul 21 02:35:52.742760 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:35:52.742731 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-mlpb8_0c38b2c7-6c2e-4f80-8789-3797e210befe/main/0.log" Jul 21 02:36:01.273897 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:01.273862 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-ljdcc_4bee3776-d7db-4be5-8e76-337ee2b89187/global-pull-secret-syncer/0.log" Jul 21 02:36:01.418573 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:01.418539 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-wj24l_e0411e85-4230-45d9-9055-1e2edf4cd322/konnectivity-agent/0.log" Jul 21 02:36:01.481724 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:01.481695 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-37.ec2.internal_03a5161441a8d59f0580132e9ff32a06/haproxy/0.log" Jul 21 02:36:06.666706 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:06.666663 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-4757q_28d4228c-c846-4f2d-b2e3-6d1c4b2298bd/manager/0.log" Jul 21 02:36:06.890984 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:06.890949 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-knw9j_c5e9de9e-2824-4d34-a651-2ad2c3d5d9ba/manager/0.log" Jul 21 02:36:08.902461 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:08.902414 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-kctq6_b393906b-0a2d-4032-b8c4-0283c219d4c9/kube-state-metrics/0.log" Jul 21 02:36:08.927391 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:08.927357 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-kctq6_b393906b-0a2d-4032-b8c4-0283c219d4c9/kube-rbac-proxy-main/0.log" Jul 21 02:36:08.951750 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:08.951724 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-kctq6_b393906b-0a2d-4032-b8c4-0283c219d4c9/kube-rbac-proxy-self/0.log" Jul 21 02:36:08.974446 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:08.974378 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7dbb746995-2zs6n_b2acc318-de88-402f-a170-17ae1f1b4ff8/metrics-server/0.log" Jul 21 02:36:09.000712 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:09.000685 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-7b7b7bc8bf-mm7rz_15b83837-cad5-45be-85d5-c8ffa9915341/monitoring-plugin/0.log" Jul 21 02:36:09.108750 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:09.108715 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6rb7p_778daaa9-d1f9-4ee5-8f24-8c291a24f197/node-exporter/0.log" Jul 21 02:36:09.130502 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:09.130474 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6rb7p_778daaa9-d1f9-4ee5-8f24-8c291a24f197/kube-rbac-proxy/0.log" Jul 21 02:36:09.153069 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:09.153045 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6rb7p_778daaa9-d1f9-4ee5-8f24-8c291a24f197/init-textfile/0.log" Jul 21 02:36:09.254052 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:09.253958 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-7llbd_0cd36c63-e03e-4496-87ae-ffaecb2434c4/kube-rbac-proxy-main/0.log" Jul 21 02:36:09.273958 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:09.273920 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-7llbd_0cd36c63-e03e-4496-87ae-ffaecb2434c4/kube-rbac-proxy-self/0.log" Jul 21 02:36:09.292701 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:09.292671 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-7llbd_0cd36c63-e03e-4496-87ae-ffaecb2434c4/openshift-state-metrics/0.log" Jul 21 02:36:09.336609 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:09.336578 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b26a8446-b697-49ec-a44f-bf5ac51e60a3/prometheus/0.log" Jul 21 02:36:09.356443 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:09.356408 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b26a8446-b697-49ec-a44f-bf5ac51e60a3/config-reloader/0.log" Jul 21 02:36:09.377417 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:09.377397 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b26a8446-b697-49ec-a44f-bf5ac51e60a3/thanos-sidecar/0.log" Jul 21 02:36:09.395652 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:09.395630 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b26a8446-b697-49ec-a44f-bf5ac51e60a3/kube-rbac-proxy-web/0.log" Jul 21 02:36:09.421937 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:09.421916 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b26a8446-b697-49ec-a44f-bf5ac51e60a3/kube-rbac-proxy/0.log" Jul 21 02:36:09.443935 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:09.443910 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b26a8446-b697-49ec-a44f-bf5ac51e60a3/kube-rbac-proxy-thanos/0.log" Jul 21 02:36:09.462636 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:09.462616 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b26a8446-b697-49ec-a44f-bf5ac51e60a3/init-config-reloader/0.log" Jul 21 02:36:09.490473 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:09.490449 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-6g4s2_ea35bdb8-4979-46f2-b79f-00ca18560d1a/prometheus-operator/0.log" Jul 21 02:36:09.504409 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:09.504338 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-6g4s2_ea35bdb8-4979-46f2-b79f-00ca18560d1a/kube-rbac-proxy/0.log" Jul 21 02:36:09.528100 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:09.528075 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-85574b5f49-7hdf7_879894ab-6a37-4e4e-8427-2d199eaaa51e/prometheus-operator-admission-webhook/0.log" Jul 21 02:36:11.069935 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:11.069897 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7t9d2/perf-node-gather-daemonset-sxz6v"] Jul 21 02:36:11.073860 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:11.073843 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7t9d2/perf-node-gather-daemonset-sxz6v" Jul 21 02:36:11.076643 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:11.076618 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7t9d2\"/\"kube-root-ca.crt\"" Jul 21 02:36:11.077637 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:11.077620 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-7t9d2\"/\"default-dockercfg-5dns8\"" Jul 21 02:36:11.077693 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:11.077675 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7t9d2\"/\"openshift-service-ca.crt\"" Jul 21 02:36:11.082618 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:11.082597 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7t9d2/perf-node-gather-daemonset-sxz6v"] Jul 21 02:36:11.144627 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:11.144596 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqcpt\" (UniqueName: \"kubernetes.io/projected/21e99c25-9c21-49e9-b329-276d9a1fef5a-kube-api-access-fqcpt\") pod \"perf-node-gather-daemonset-sxz6v\" (UID: \"21e99c25-9c21-49e9-b329-276d9a1fef5a\") " pod="openshift-must-gather-7t9d2/perf-node-gather-daemonset-sxz6v" Jul 21 02:36:11.144793 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:11.144644 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/21e99c25-9c21-49e9-b329-276d9a1fef5a-lib-modules\") pod \"perf-node-gather-daemonset-sxz6v\" (UID: \"21e99c25-9c21-49e9-b329-276d9a1fef5a\") " pod="openshift-must-gather-7t9d2/perf-node-gather-daemonset-sxz6v" Jul 21 02:36:11.144793 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:11.144745 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/21e99c25-9c21-49e9-b329-276d9a1fef5a-podres\") pod \"perf-node-gather-daemonset-sxz6v\" (UID: \"21e99c25-9c21-49e9-b329-276d9a1fef5a\") " pod="openshift-must-gather-7t9d2/perf-node-gather-daemonset-sxz6v" Jul 21 02:36:11.144793 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:11.144784 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/21e99c25-9c21-49e9-b329-276d9a1fef5a-sys\") pod \"perf-node-gather-daemonset-sxz6v\" (UID: \"21e99c25-9c21-49e9-b329-276d9a1fef5a\") " pod="openshift-must-gather-7t9d2/perf-node-gather-daemonset-sxz6v" Jul 21 02:36:11.144895 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:11.144800 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/21e99c25-9c21-49e9-b329-276d9a1fef5a-proc\") pod \"perf-node-gather-daemonset-sxz6v\" (UID: \"21e99c25-9c21-49e9-b329-276d9a1fef5a\") " pod="openshift-must-gather-7t9d2/perf-node-gather-daemonset-sxz6v" Jul 21 02:36:11.246102 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:11.246073 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/21e99c25-9c21-49e9-b329-276d9a1fef5a-podres\") pod \"perf-node-gather-daemonset-sxz6v\" (UID: \"21e99c25-9c21-49e9-b329-276d9a1fef5a\") " pod="openshift-must-gather-7t9d2/perf-node-gather-daemonset-sxz6v" Jul 21 02:36:11.246281 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:11.246121 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/21e99c25-9c21-49e9-b329-276d9a1fef5a-sys\") pod \"perf-node-gather-daemonset-sxz6v\" (UID: \"21e99c25-9c21-49e9-b329-276d9a1fef5a\") " pod="openshift-must-gather-7t9d2/perf-node-gather-daemonset-sxz6v" Jul 21 02:36:11.246281 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:11.246144 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/21e99c25-9c21-49e9-b329-276d9a1fef5a-proc\") pod \"perf-node-gather-daemonset-sxz6v\" (UID: \"21e99c25-9c21-49e9-b329-276d9a1fef5a\") " pod="openshift-must-gather-7t9d2/perf-node-gather-daemonset-sxz6v" Jul 21 02:36:11.246281 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:11.246195 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqcpt\" (UniqueName: \"kubernetes.io/projected/21e99c25-9c21-49e9-b329-276d9a1fef5a-kube-api-access-fqcpt\") pod \"perf-node-gather-daemonset-sxz6v\" (UID: \"21e99c25-9c21-49e9-b329-276d9a1fef5a\") " pod="openshift-must-gather-7t9d2/perf-node-gather-daemonset-sxz6v" Jul 21 02:36:11.246281 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:11.246213 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/21e99c25-9c21-49e9-b329-276d9a1fef5a-sys\") pod \"perf-node-gather-daemonset-sxz6v\" (UID: \"21e99c25-9c21-49e9-b329-276d9a1fef5a\") " pod="openshift-must-gather-7t9d2/perf-node-gather-daemonset-sxz6v" Jul 21 02:36:11.246281 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:11.246241 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/21e99c25-9c21-49e9-b329-276d9a1fef5a-podres\") pod \"perf-node-gather-daemonset-sxz6v\" (UID: \"21e99c25-9c21-49e9-b329-276d9a1fef5a\") " pod="openshift-must-gather-7t9d2/perf-node-gather-daemonset-sxz6v" Jul 21 02:36:11.246281 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:11.246252 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/21e99c25-9c21-49e9-b329-276d9a1fef5a-lib-modules\") pod \"perf-node-gather-daemonset-sxz6v\" (UID: \"21e99c25-9c21-49e9-b329-276d9a1fef5a\") " pod="openshift-must-gather-7t9d2/perf-node-gather-daemonset-sxz6v" Jul 21 02:36:11.246590 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:11.246292 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/21e99c25-9c21-49e9-b329-276d9a1fef5a-proc\") pod \"perf-node-gather-daemonset-sxz6v\" (UID: \"21e99c25-9c21-49e9-b329-276d9a1fef5a\") " pod="openshift-must-gather-7t9d2/perf-node-gather-daemonset-sxz6v" Jul 21 02:36:11.246590 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:11.246365 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/21e99c25-9c21-49e9-b329-276d9a1fef5a-lib-modules\") pod \"perf-node-gather-daemonset-sxz6v\" (UID: \"21e99c25-9c21-49e9-b329-276d9a1fef5a\") " pod="openshift-must-gather-7t9d2/perf-node-gather-daemonset-sxz6v" Jul 21 02:36:11.254538 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:11.254509 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqcpt\" (UniqueName: \"kubernetes.io/projected/21e99c25-9c21-49e9-b329-276d9a1fef5a-kube-api-access-fqcpt\") pod \"perf-node-gather-daemonset-sxz6v\" (UID: \"21e99c25-9c21-49e9-b329-276d9a1fef5a\") " pod="openshift-must-gather-7t9d2/perf-node-gather-daemonset-sxz6v" Jul 21 02:36:11.385214 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:11.385129 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7t9d2/perf-node-gather-daemonset-sxz6v" Jul 21 02:36:11.518649 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:11.518620 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7t9d2/perf-node-gather-daemonset-sxz6v"] Jul 21 02:36:11.521137 ip-10-0-141-37 kubenswrapper[2574]: W0721 02:36:11.521111 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod21e99c25_9c21_49e9_b329_276d9a1fef5a.slice/crio-b74f980937edc7593c39c70d6970f21f03ac4f6e80d503f956742fc65cbccc62 WatchSource:0}: Error finding container b74f980937edc7593c39c70d6970f21f03ac4f6e80d503f956742fc65cbccc62: Status 404 returned error can't find the container with id b74f980937edc7593c39c70d6970f21f03ac4f6e80d503f956742fc65cbccc62 Jul 21 02:36:12.044939 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:12.044895 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-5bc67dcd46-psp69_8fabdf69-8196-42e4-9ffc-5f50f6eef7cd/download-server/0.log" Jul 21 02:36:12.428381 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:12.428302 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7t9d2/perf-node-gather-daemonset-sxz6v" event={"ID":"21e99c25-9c21-49e9-b329-276d9a1fef5a","Type":"ContainerStarted","Data":"a691766f423f0bdb70151f5cbbf4e8a527ebfc10b56ec6fee7ee68aa3069fa6c"} Jul 21 02:36:12.428381 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:12.428354 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7t9d2/perf-node-gather-daemonset-sxz6v" event={"ID":"21e99c25-9c21-49e9-b329-276d9a1fef5a","Type":"ContainerStarted","Data":"b74f980937edc7593c39c70d6970f21f03ac4f6e80d503f956742fc65cbccc62"} Jul 21 02:36:12.428763 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:12.428385 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-7t9d2/perf-node-gather-daemonset-sxz6v" Jul 21 02:36:12.446909 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:12.446859 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-7t9d2/perf-node-gather-daemonset-sxz6v" podStartSLOduration=1.446844732 podStartE2EDuration="1.446844732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 02:36:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:36:12.44409979 +0000 UTC m=+1606.463593696" watchObservedRunningTime="2026-07-21 02:36:12.446844732 +0000 UTC m=+1606.466338692" Jul 21 02:36:13.566515 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:13.566483 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-24k47_f7a19bce-85cf-4f26-aafa-977786cbfc4d/dns/0.log" Jul 21 02:36:13.586190 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:13.586162 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-24k47_f7a19bce-85cf-4f26-aafa-977786cbfc4d/kube-rbac-proxy/0.log" Jul 21 02:36:13.684817 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:13.684791 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8d69j_c8e442c1-e98a-4af5-9274-5ab79f671aef/dns-node-resolver/0.log" Jul 21 02:36:14.247567 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:14.247535 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7bc6c6857c-7rj6n_a56f6c6c-6e08-41fc-b8ad-69c9fb0f82e3/registry/0.log" Jul 21 02:36:14.287092 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:14.287067 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-b9rfh_83fe72d8-5aa8-4232-9317-3e0d06bc9535/node-ca/0.log" Jul 21 02:36:15.285939 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:15.285900 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-2zbzn_9cbeff22-f3f1-4e40-85c8-02c7971c9c47/discovery/0.log" Jul 21 02:36:15.304575 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:15.304545 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d4c7dc87d-52tfh_e2d7f3ea-52ce-4b44-9821-6f34b323f0f9/kube-auth-proxy/0.log" Jul 21 02:36:15.433890 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:15.433858 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-222pw_ed4564f1-d28f-471e-9f52-c2fb73e5543d/payload-pre-processing/0.log" Jul 21 02:36:15.465603 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:15.465571 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-774v7_ba7a6633-133a-4815-8cda-5653df7c9535/payload-processing/0.log" Jul 21 02:36:16.084412 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:16.084374 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pc4fb_224189a4-715f-4007-8d45-832188c1f1d1/serve-healthcheck-canary/0.log" Jul 21 02:36:16.830850 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:16.830821 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ncxvc_a89ab6c0-0bd2-4cb5-aca9-71b07c462005/kube-rbac-proxy/0.log" Jul 21 02:36:16.850019 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:16.849995 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ncxvc_a89ab6c0-0bd2-4cb5-aca9-71b07c462005/exporter/0.log" Jul 21 02:36:16.869743 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:16.869713 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ncxvc_a89ab6c0-0bd2-4cb5-aca9-71b07c462005/extractor/0.log" Jul 21 02:36:18.443772 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:18.443683 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-7t9d2/perf-node-gather-daemonset-sxz6v" Jul 21 02:36:19.038258 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:19.038202 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-9x9k7_b0e21853-e493-4e53-9b6f-fdc88f1d766f/manager/0.log" Jul 21 02:36:19.153095 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:19.153063 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-pwzb2_54ddeee7-7991-44cb-9c3a-dc445f966bb2/manager/0.log" Jul 21 02:36:19.253879 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:19.253850 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-mjtvp_7126c66e-8b6b-4a9b-b93f-62f9d393ad72/server/0.log" Jul 21 02:36:19.325186 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:19.325101 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-2pth6_21493e6e-e2a6-41cc-8483-a644130919a2/manager/0.log" Jul 21 02:36:19.358148 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:19.358115 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8475997445-2dwbj_c1c91659-2ab2-4743-a85f-648bf5b5f5b8/manager/0.log" Jul 21 02:36:19.380532 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:19.380512 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8475997445-2dwbj_c1c91659-2ab2-4743-a85f-648bf5b5f5b8/copy-manifests/0.log" Jul 21 02:36:21.246032 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:21.245992 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-2zbzn_9cbeff22-f3f1-4e40-85c8-02c7971c9c47/discovery/0.log" Jul 21 02:36:21.254100 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:21.254071 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d4c7dc87d-52tfh_e2d7f3ea-52ce-4b44-9821-6f34b323f0f9/kube-auth-proxy/0.log" Jul 21 02:36:21.359552 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:21.359475 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-222pw_ed4564f1-d28f-471e-9f52-c2fb73e5543d/payload-pre-processing/0.log" Jul 21 02:36:21.386577 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:21.386541 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-774v7_ba7a6633-133a-4815-8cda-5653df7c9535/payload-processing/0.log" Jul 21 02:36:22.477542 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:22.477512 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-2zbzn_9cbeff22-f3f1-4e40-85c8-02c7971c9c47/discovery/0.log" Jul 21 02:36:22.491646 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:22.491624 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d4c7dc87d-52tfh_e2d7f3ea-52ce-4b44-9821-6f34b323f0f9/kube-auth-proxy/0.log" Jul 21 02:36:22.613690 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:22.613606 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-222pw_ed4564f1-d28f-471e-9f52-c2fb73e5543d/payload-pre-processing/0.log" Jul 21 02:36:22.639815 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:22.639787 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-774v7_ba7a6633-133a-4815-8cda-5653df7c9535/payload-processing/0.log" Jul 21 02:36:27.720792 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:27.720761 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-x2w8b_3504ec09-d4b6-41ce-b77d-d6c73b437a33/kube-multus-additional-cni-plugins/0.log" Jul 21 02:36:27.735695 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:27.735672 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-x2w8b_3504ec09-d4b6-41ce-b77d-d6c73b437a33/egress-router-binary-copy/0.log" Jul 21 02:36:27.755244 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:27.755220 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-x2w8b_3504ec09-d4b6-41ce-b77d-d6c73b437a33/cni-plugins/0.log" Jul 21 02:36:27.770145 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:27.770123 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-x2w8b_3504ec09-d4b6-41ce-b77d-d6c73b437a33/bond-cni-plugin/0.log" Jul 21 02:36:27.787780 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:27.787764 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-x2w8b_3504ec09-d4b6-41ce-b77d-d6c73b437a33/routeoverride-cni/0.log" Jul 21 02:36:27.804642 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:27.804621 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-x2w8b_3504ec09-d4b6-41ce-b77d-d6c73b437a33/whereabouts-cni-bincopy/0.log" Jul 21 02:36:27.821254 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:27.821232 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-x2w8b_3504ec09-d4b6-41ce-b77d-d6c73b437a33/whereabouts-cni/0.log" Jul 21 02:36:27.855798 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:27.855772 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-lkmsc_51277326-7122-47d0-991c-4fe0c5aa0102/kube-multus/0.log" Jul 21 02:36:27.932075 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:27.932035 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-8v67q_03e524b0-40bd-46b4-ac8f-eb346e94a7dc/network-metrics-daemon/0.log" Jul 21 02:36:27.944597 ip-10-0-141-37 kubenswrapper[2574]: I0721 02:36:27.944563 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-8v67q_03e524b0-40bd-46b4-ac8f-eb346e94a7dc/kube-rbac-proxy/0.log"